Skip to content

Revert "topologies: switch all .m4 files to codec_provider and codec_consumer#5192

Merged
lgirdwood merged 1 commit into
thesofproject:mainfrom
marc-hb:revert-consumer
Feb 18, 2022
Merged

Revert "topologies: switch all .m4 files to codec_provider and codec_consumer#5192
lgirdwood merged 1 commit into
thesofproject:mainfrom
marc-hb:revert-consumer

Conversation

@marc-hb
Copy link
Copy Markdown
Collaborator

@marc-hb marc-hb commented Jan 12, 2022

This reverts commit f50b6fe.

I discovered the hard way that this change causes alsatplg version
1.2.2 (the default version in the current LTS Ubuntu) to corrupt just
a few bytes in .tplg output in an incredibly discrete and
time-consuming way: no error message at build time and same file size.
See example at #5162.

We could/should just require a minimum alsatplg version at the CMake
level but at this moment we don't even know which minimum version is
needed and we would also need to take some time to test a few alsatplg
versions. If version 1.2.2 would just fail with an decent error
message that can be searched and discussed then everything would be
fine but silent corruption is really not OK.

So users of recent versions will unfortunately have to live with the
huge number of warnings for now.

Signed-off-by: Marc Herbert marc.herbert@intel.com

@aiChaoSONG

This comment was marked as off-topic.

@marc-hb

This comment was marked as off-topic.

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

I spent more time looking at the recursive diffoscope output (awesome tool, use it). What happens is very simple:

             codec_slave  codec_consumer
1.2.2        0101                0000
1.2.6        0101                0101

Example:

├── sof-acp.conf
@@ -844,9 +844,9 @@
 	format          "I2S"
        mclk              "codec_mclk_in"
         mclk_freq         "49152000"
-       bclk              "codec_consumer"
+       bclk              "codec_slave"
         bclk_freq         "3072000"
-       fsync              "codec_consumer"
+       fsync              "codec_slave"
         fsync_freq         "48000"
               tdm_slots       "2"
        tdm_slot_width  "32"
├── sof-acp.tplg
 000015d0  00 00 00 00 00 00 00 78  00 00 00 00 00 00 00 01  |.......x........|
-000015e0  00 00 00 00 00 00 01 01  01 00 00 00 00 ee 02 00  |................|
+000015e0  00 00 00 00 00 00 00 00  01 00 00 00 00 ee 02 00  |................|
 000015f0  e0 2e 00 80 bb 00 00 02  00 00 00 20 00 00 00 03  |........... ....|
 00001600  00 00 00 03 00 00 00 00  00 00 00 00 00 00 00 00  |................|
 00001610  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

All topologies show the same difference.

@marc-hb marc-hb requested a review from perexg January 12, 2022 02:40
@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

Copy link
Copy Markdown
Collaborator

@aiChaoSONG aiChaoSONG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inclusive words codec_consumer and codec_provider are added after the alsa-lib 1.2.2 by Pierre. With #5162, the alsa-lib 1.2.2 cannot recognize the inclusive words, so the code goes to the ELSE part and will assign wrong value to bclk_provider(was bclk_master) and fsync(was fsync_master).

#5162 should be merged into the code once alsa-lib 1.2.2 becomes deprecated and no one uses it, but not now.

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

Thanks @aiChaoSONG ! So every alsatplg version before https://github.com/alsa-project/alsa-lib/commits/706192341d1d0bbb906 will silently corrupt the .tplg output, correct? If yes did you find (maybe with git describe --contains) which is the first alsatplg release after https://github.com/alsa-project/alsa-lib/commits/706192341d1d0bbb906 ?

The alternative to this revert is to (loudly) fail the build and force developers to upgrade ALSA to the minimum version required. @ranj063 recently added such a version check for topology2, we could easily add another one.

@aiChaoSONG
Copy link
Copy Markdown
Collaborator

@marc-hb It's alsa-lib 1.2.5

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

@marc-hb It's alsa-lib 1.2.5

So I think 1.2.5 is too recent and we should merge this revert.

While Fedora is tracking ALSA very closely thanks to @perexg , other distributions are lagging behind
https://packages.fedoraproject.org/pkgs/alsa-lib/alsa-lib/

The latest Ubuntu 21.10 is still on 1.2.4 (the next one will have at least 1.2.5)
https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libatopology&searchon=names

The latest Debian release is less than one month old and it still has 1.2.4
https://packages.debian.org/bullseye/libatopology2

https://build.opensuse.org/request/show/899947 upgrade to 1.2.5 in June 2021, dunno which release that maps to.

(1.2.5 was apparently tagged in May 2021)

@perexg
Copy link
Copy Markdown

perexg commented Jan 12, 2022

It not easier to check the alsatplg version before the call (in the build system) rather than to revert the whole change?

Also, the code should be improved to fail when an unknown value is set.

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

It not easier to check the alsatplg version before the call (in the build system) rather than to revert the whole change?

Considered above.

Also, the code should be improved to fail when an unknown value is set.

Which code? I don't think any distro would make any emergency 1.2.4.1 ALSA upgrade just for this.

This rename is simply too early, that's all.

When made this rename I realized the corresponding ALSA change was only 1 year old. However:

  • I underestimated the release and distro delays
  • much more importantly, I didn't expect any silent corruption (I should have considering it's all in C)

It's unfortunate the deprecation warning was added at the same time than the feature without any transition period.

@perexg
Copy link
Copy Markdown

perexg commented Jan 12, 2022

It not easier to check the alsatplg version before the call (in the build system) rather than to revert the whole change?

Considered above.

It looks like a bit bad workflow if you don't run your tests against the latest alsa-utils (alsatplg). The code is in upstream really long time enough for the verification and you can do the verification right now.

Alternate way is to replace the clock strings in the configuration files back for the older alsatplg binaries (use sed in the build system or so).

Also, the code should be improved to fail when an unknown value is set.

Which code? I don't think any distro would make any emergency 1.2.4.1 ALSA upgrade just for this.

The upstream code to avoid such situations in future.

@perexg
Copy link
Copy Markdown

perexg commented Jan 12, 2022

Note - I believe that the generated ,tplg binaries between alsatplg 1.2.2 and recent should be identical (so the verification can compare the .tlpg binaries only - no hw tests are required). It should be a quick job.

@lgirdwood
Copy link
Copy Markdown
Member

@ranj063 can you confirm the topology2 fails for unknown values ?
@marc-hb any reason why we are using the older ALSA releases in our validation ? We should be using the latest version that will ship with current Fedora/Ubuntu. I dont think there is any point testing LTS when newer non LTS releaese are current (as LTS will probably never upgrade ALSA, FW or kernel releases AFAICT).

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 12, 2022

can you confirm the topology2 fails for unknown values ?

I think we're not really using topology2 anywhere yet.

any reason why we are using the older ALSA releases in our validation ?

Validation is using ALSA 1.2.6 now, this revert is for developers.

It looks like a bit bad workflow if you don't run your tests against the latest alsa-utils (alsatplg).

Yes it's "bad" - does it have to be "punished" with silent .tplg corruption and very obscure errors? Do we think we have too many SOF developers, is this how we want to welcome new ones? I was lucky to identify this issue in less than a couple hours only because I rebase every day and could bisect quickly. I also made the change in the first place.

Having multiple library versions installed concurrently on Linux is a real pain, if the distro one "seems" to work it's not surprising people stick to it.

The upstream code to avoid such situations in future.

+1 and thanks for the other ideas.

@perexg
Copy link
Copy Markdown

perexg commented Jan 12, 2022

any reason why we are using the older ALSA releases in our validation ?

Validation is using ALSA 1.2.6 now, this revert is for developers.

In this case, the developers should be informed to use the new ALSA release (and the build system should fail). It's quite easy to develop things using docker/podman, if they want to use the distro without regular upstream updates. Only my two cents. I won't comment this more.

Copy link
Copy Markdown
Contributor

@greg-intel greg-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approved the original, and am disappointed that this didn't work out.
The warnings are very noisy.

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 19, 2022

Good developer experience is entirely about removing as much friction as possible.

From https://thesofproject.github.io/latest/getting_started/build-guide/build-from-scratch.html?highlight=alsa#build-alsa-lib-and-alsa-utils-from-source

Installing alsa-lib systemwide may break some audio applications. Only perform this if you know what you are doing. We recommend that you install it locally (under $HOME) or use Docker (see Build SOF with Docker.)

Compiling gcc toolchains is also a huge amount of friction but only once.

Docker is relatively easy to use but it's not frictionless either. For instance:
https://linuxnatives.net/2019/align-user-ids-inside-and-outside-docker-with-subuser-mapping
https://medium.com/@mccode/the-misunderstood-docker-tag-latest-af3babfd6375
etc.

In fact in related #5202 I asked some (experienced) developers to test with https://thesofproject.github.io/latest/getting_started/build-guide/build-with-docker.html#build-firmware-binaries-with-docker and even when these instructions are very short, those who have never used docker yet were... not interested! Friction. https://www.google.com/search?q=friction+habits

@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Jan 24, 2022

While Fedora is tracking ALSA very closely thanks to @perexg , other distributions are lagging behind
https://packages.fedoraproject.org/pkgs/alsa-lib/alsa-lib/

Actually, even Fedora 35 does not work out of the box. It's supposed to work:

rpm -qa alsa*
alsa-lib-1.2.6.1-3.fc35.x86_64
alsa-topology-1.2.6.1-3.fc35.noarch
alsa-ucm-1.2.6.1-3.fc35.noarch
alsa-firmware-1.2.4-5.fc35.noarch
alsa-tools-firmware-1.2.5-2.fc35.x86_64
alsa-utils-1.2.6-1.fc35.x86_64
alsa-lib-devel-1.2.6.1-3.fc35.x86_64
alsa-topology-utils-1.2.6-1.fc35.x86_64
alsa-lib-1.2.6.1-3.fc35.i686
alsa-sof-firmware-1.9.3-1.fc35.noarch

Yet the -I, --inc-dir=DIR is missing from some unknown reason:

alsatplg --help
Usage: alsatplg [OPTIONS]...

-h, --help              help
-c, --compile=FILE      compile configuration file
-p, --pre-process       pre-process Topology2.0 configuration file before compilation
-P, --pre-process=FILE  pre-process Topology2.0 configuration file
-d, --decode=FILE       decode binary topology file
-n, --normalize=FILE    normalize configuration file
-u, --dump=FILE         dump (reparse) configuration file
-v, --verbose=LEVEL     set verbosity level (0...1)
-o, --output=FILE       set output file
-D, --define=ARGS       define variables (VAR1=VAL1[,VAR2=VAL2] ...)
-s, --sort              sort the identifiers in the normalized output
-g, --group             save configuration by group indexes
-x, --nocheck           save configuration without additional integrity checks
-z, --dapm-nosort       do not sort the DAPM widgets
-V, --version           print version

So ./scripts/build-tools.sh -T fails with a cryptic:

Generating cavs-gain-hda.tplg
alsatplg: invalid option -- 'I'

What a poor experience.

I found this suspicious bit https://github.com/alsa-project/alsa-utils/blob/40202a522aa86599bffaabf8097bf9bf1cef4f61/topology/topology.c#L59

"-o, --output=FILE       set output file\n"
#if SND_LIB_VER(1, 2, 5) < SND_LIB_VERSION
"-D, --define=ARGS       define variables (VAR1=VAL1[,VAR2=VAL2] ...)\n"
"-I, --inc-dir=DIR       set include path\n"
#endif
"-s, --sort              sort the identifiers in the normalized output\n"

(funny how the default branch names at https://github.com/alsa-project and https://git.alsa-project.org/ are still "master")

…consumer"

This reverts commit f50b6fe.

I discovered the hard way that this change causes alsatplg version
1.2.2 (the default version in the current LTS Ubuntu) to corrupt just
a few bytes in .tplg output in an incredibly discrete and
time-consuming way: no error message at build time and same file size.
See example at thesofproject#5162.

We could/should just require a minimum alsatplg version at the CMake
level but at this moment we don't even know which minimum version is
needed and we would also need to take some time to test a few alsatplg
versions. If version 1.2.2 would just fail with an decent error
message that can be searched and discussed then everything would be
fine but silent corruption is really not OK.

So users of recent versions will unfortunately have to live with the
huge number of warnings for now.

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Feb 16, 2022

@lgirdwood lgirdwood merged commit 15a7774 into thesofproject:main Feb 18, 2022
@marc-hb marc-hb deleted the revert-consumer branch March 14, 2022 23:47
marc-hb added a commit to marc-hb/sof that referenced this pull request Oct 19, 2023
We'll use this new function to require ALSA 1.2.5 and finally rename to
"codec_consumer" (see revert discussion thesofproject#5192)

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
marc-hb added a commit to marc-hb/sof that referenced this pull request Oct 19, 2023
We'll use this new function to require ALSA 1.2.5 and finally rename to
"codec_consumer" (see revert discussion thesofproject#5192)

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
lgirdwood pushed a commit that referenced this pull request Oct 23, 2023
We'll use this new function to require ALSA 1.2.5 and finally rename to
"codec_consumer" (see revert discussion #5192)

Signed-off-by: Marc Herbert <marc.herbert@intel.com>
@marc-hb
Copy link
Copy Markdown
Collaborator Author

marc-hb commented Oct 23, 2023

Note another, unrelated and much more recent silent corruption with alsatplg:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants