Skip to content

seth/rails-7.2#60437

Draft
snickell wants to merge 157 commits into
stagingfrom
seth/rails-7.2
Draft

seth/rails-7.2#60437
snickell wants to merge 157 commits into
stagingfrom
seth/rails-7.2

Conversation

@snickell

@snickell snickell commented Aug 15, 2024

Copy link
Copy Markdown
Collaborator

Follow-on to the Rails 3.3.4 upgrade PR, this PR upgrades Rails to 7.2.

  • Read changelogs/git-commits of Rails-related gems, and upgrade those who's changelog mentions changes to support Rails 7.x.
  • bundle update rails .... keep adding to the list until no Gem conflicts.
  • Disable composite_primary_keys gem, as support is now present in Rails as of 7.1. See: https://www.honeybadger.io/blog/composite-keys-in-rails/
  • Disable mysql_check_index_used.rb, at least temporarily, see: related comment

TODO

  • application.rb: # config.action_mailer.delivery_job = 'MailDeliveryJob', see seth/rails-7.2 #60437 (comment)
  • initializers/backtrace_silencers.rb: remove, see seth/rails-7.2 #60437 (comment)
  • Check how rails 7's built-in composite primary key support works with the annotate gem (we are using the latest version, this may change some model headers, it doesn't mark primary composite keys as primary in each item).
  • Decide if we want to remove mysql_check_index_used.rb, see: related comment

snickell added 30 commits August 9, 2024 23:12
… anyway) which was spewing warnings to console
…undle version (bundle does this automatically to match Gemfile.lock bundled with on bundle install)
…m to 3.5.17

Gem is an interesting case, the default gem version with ruby with 3.3.4 is 3.5.11, but a previous update of default.rb by Elijah had trouble unless a version of rubygem was specified that was NOT the system version, see: bf41ebf
…bled = no verification)

cdo-ruby/ ruby_spec.rb: verify versions
…r chef will ignore if files already exist from a previous version
… "000000" were turning into the int 0 which confuses the poor badger
… but will no longer be part of the default gems since Ruby 3.4.0"
@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author
/Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/attribute_methods/serialization.rb:183:in `serialize': wrong number of arguments (given 2, expected 1) (ArgumentError)
        from /Users/seth/src/code-dot-org/dashboard/app/models/sections/section.rb:96:in `<class:Section>'
        from /Users/seth/src/code-dot-org/dashboard/app/models/sections/section.rb:42:in `<top (required)>'

Fix is to explicitly pass coder: to serialize as a keyword arg.

Fixed by: d33d7e9

@snickell

Copy link
Copy Markdown
Collaborator Author

migrating off composite_primary_key to rails built-in

/Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/dynamic_matchers.rb:22:in `method_missing': undefined method `primary_keys=' for class DatablockStorageTable (NoMethodError)
        from /Users/seth/src/code-dot-org/dashboard/app/models/datablock_storage_table.rb:32:in `<class:DatablockStorageTable>'

Fixed by: 7a8d698

@snickell

Copy link
Copy Markdown
Collaborator Author
/Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/mocha-1.1.0/lib/mocha/integration/mini_test/adapter.rb:27:in `included': uninitialized constant MiniTest (NameError)

          Mocha::ExpectationErrorFactory.exception_class = ::MiniTest::Assertion
                                                           ^^^^^^^^^^
Did you mean?  Minitest
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/mocha-1.1.0/lib/mocha/integration/mini_test.rb:43:in `include'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/mocha-1.1.0/lib/mocha/integration/mini_test.rb:43:in `activate'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/mocha-1.1.0/lib/mocha/mini_test.rb:3:in `<main>'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/kernel.rb:34:in `require'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /Users/seth/src/code-dot-org/dashboard/test/test_helper.rb:25:in `<main>'

Closely related issue: freerange/mocha#614

This could be fixed by upgrading mocha, but its a major version jump and seems like it could open a can of worms. Instead I've opted to work around this by pulling in the old MiniTest constants using require 'minitest/unit' which contains a compat layer.

Upgrading mocha+minitest could be revisited in a follow-up PR, probably a good idea anyway.

Fixed by: 3e7344a

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author
bundled_gems.rb:74:in `require': cannot load such file -- i18n_controller (LoadError)
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/kernel.rb:34:in `require'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
        from /Users/seth/src/code-dot-org/dashboard/test/controllers/i18n_controller_test.rb:2:in `<main>'

Another require that should be loaded by zeitwerk automatically.

Fixed by: 234a3b3

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author

Remove lock_thread.rb, a "feature backport from Rails 5.1"

code-dot-org/dashboard/test/testing/lock_thread.rb:25:in `with_connection': undefined method `[]' for nil (NoMethodError)

    unless (conn = @thread_cached_conns[owner_thread])
                                       ^^^^^^^^^^^^^^
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/connection_handling.rb:296:in `with_connection'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/relation/calculations.rb:317:in `block in pluck'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/relation.rb:1452:in `skip_query_cache_if_necessary'
        from /Users/seth/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/activerecord-7.2.0/lib/active_record/relation/calculations.rb:313:in `pluck'
        from /Users/seth/src/code-dot-org/dashboard/app/models/unit.rb:861:in `unit_names_by_curriculum_umbrella'

This was fixed in rails here: rails/rails#28083

We backported the feature, but the backport no longer works and would appear to be redundant in modern rails.

If all the tests break in crazy non-deterministic ways, this might be a good comment to revisit ;-)

Removed by: 6dbd485

Fix transactional_test_case.rb

ERROR["teardown_all", Pd::InternationalOptInControllerTest, 7.714712999761105]
 teardown_all#Pd::InternationalOptInControllerTest (7.71s)
Minitest::UnexpectedError:         NoMethodError: undefined method `lock_thread=' for an instance of ActiveRecord::ConnectionAdapters::ConnectionPool
            test/testing/transactional_test_case.rb:41:in `block (3 levels) in <module:TransactionalTestCase>'
            test/testing/transactional_test_case.rb:39:in `each'
            test/testing/transactional_test_case.rb:39:in `block (2 levels) in <module:TransactionalTestCase>'
            test/testing/setup_all_and_teardown_all.rb:54:in `block (2 levels) in run_callbacks'
            test/testing/setup_all_and_teardown_all.rb:53:in `block in run_callbacks'
            test/testing/setup_all_and_teardown_all.rb:52:in `run_callbacks'
            test/testing/setup_all_and_teardown_all.rb:37:in `run'

Rails 7.2 removes def lock_thread= from connection_pool.rb in rails/rails#50999, replacing it with def pin_connection! and def unpin_connection!, which ALSO manage the transaction creation/rollback, fortunately in a way that exactly mirrors what we have in transactional_test_case.rb. As a result, I've replaced lock_thread= with a call to pin_connection!, as well as opening and rolling back the transaction.

Fixed by: de3a5f8

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author

Tests now start to run, but ALL fail with errors exactly like:

ERROR["test_deleted_user_cannot_sign_in", "SessionsControllerTest", 0.0067719994112849236]
 test_deleted_user_cannot_sign_in#SessionsControllerTest (0.01s)
Minitest::UnexpectedError:         NoMethodError: undefined method `fetch' for nil
            test/testing/setup_all_and_teardown_all.rb:36:in `run'

Repros with: bundle exec spring testunit ./test/models/lesson_activity_test.rb

Well this was a crazy one and took almost an hour to find. This bizarre error stems from not having a config/cable.yml file. ActionCable::TestCase is in the inheritance hierarchy for TestCase now, and it must try to load this file in its init. Solution is to paste in a commented out copy of the stock config/cable.yml from a fresh Rails 7.2 app. Hopefully we'll be turning this on soon, but not today ;-)

Fixed by: 5243242

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author

A bunch of LevelsHelperTest based tests failing with this backtrace:

ERROR["test_app_options_sets_level_requires_channel_to_true_if_level_is_channel_backed", "LevelsHelperTest", 1.076027000322938]
 test_app_options_sets_level_requires_channel_to_true_if_level_is_channel_backed#LevelsHelperTest (1.08s)
Minitest::UnexpectedError:         ArgumentError: unknown keyword: :database
            app/helpers/read_replica_helper.rb:21:in `connected_to'
            app/models/channel_token.rb:47:in `find_or_create_channel_token'
            app/helpers/levels_helper.rb:99:in `get_channel_for'
            app/helpers/levels_helper.rb:213:in `app_options'
            test/helpers/levels_helper_test.rb:308:in `block in <class:LevelsHelperTest>'
            test/testing/setup_all_and_teardown_all.rb:36:in `run'

Repros locally with: bundle exec spring testunit ./test/helpers/levels_helper_test.rb

In Rails 6.0.1 connected_to has this signature:
connected_to(database: nil, role: nil, prevent_writes: false, &blk)

In Rails 7.2 connected_to has this signature:
connected_to(role: nil, shard: nil, prevent_writes: false, &blk)

Fix is to modify connected_to in read_replica_helper.rb to neither take nor pass on a database: keyword arg.

Fixed by: 0faac07

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author

WARN: Multiple ActiveRecord connections are in use

Still getting lots of warnings:

WARN: Multiple ActiveRecord connections are in use, this can make transactional tests fail in weird ways.

Repros with: bundle exec spring testunit ./test/controllers/api/v1/pd/teacher_attendance_report_controller_test.rb

in transactional_test_case.rb we see:

        setup do
          pools = ActiveRecord::Base.connection_handler.connection_pool_list
          all_connections = pools.map(&:connections).flatten
          if all_connections.many?
            warning = 'WARN: Multiple ActiveRecord connections are in use, this can make transactional tests fail in weird ways.'
            CDO.log.warn warning
            puts warning
        end

Rails 7.2

In-situ

If we inspect all_connections and pools in-situ with binding.irb:

[test] dashboard > all_connections
=> [#<ActiveRecord::ConnectionAdapters::Mysql2Adapter:0x00000000134d18 env_name="test" role=:writing>, 
#<ActiveRecord::ConnectionAdapters::Mysql2Adapter:0x00000000134d18 env_name="test" role=:writing>]
[test] dashboard > ActiveRecord::Base.connection_handler.connection_pool_list
=> [#<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="test" role=:writing>, 
#<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="test" role=:writing>]

In other words, we have multiple connections because ActiveRecord::Base.connection_handler.connection_pool_list is returning two pools, both with the writing role connection in them.

rails console

What's interesting is that if we load rails console, we see more what we'd expect: one reading pool and one writing pool:

[test] dashboard > ActiveRecord::Base.connection_handler.connection_pool_list
=> 
[#<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="test" role=:writing>,
 #<ActiveRecord::ConnectionAdapters::ConnectionPool env_name="test" role=:reading>]

Rails 6.1.7.7

in-situ

[test] dashboard > pools.length
=> 1

rails console

[test] dashboard > ActiveRecord::Base.connection_handler.connection_pool_list.length
=> 1

Fix

We hardcode our read/write replicas (following the pattern from a medium article) in active_record_roles.rb and embedded in application.rb like so:

  connects_to database: {
    writing: Policies::ActiveRecordRoles.get_writing_role_name, # => :primary in test
    reading: Policies::ActiveRecordRoles.get_reading_role_name # => also :primary in test
  }

Even when these are both the same connection pool, Rails 7.2 lists the connection pool twice. As a result, we trigger the warning. The fix is to de-duplicate our pools list. This is safe because we lock at the pool level.

Fixed by: 619a8fc

@snickell

snickell commented Aug 22, 2024

Copy link
Copy Markdown
Collaborator Author

alias_attribute doesn't work for associations anymore

ERROR["test_not_logged-in_user_calling_get_show_by_keys_should_receive_success_to_", "ProgrammingClassesControllerTest::ProgrammingClassesControllerAccessTests", 202.97481100074947]
 test_not_logged-in_user_calling_get_show_by_keys_should_receive_success_to_#ProgrammingClassesControllerTest::ProgrammingClassesControllerAccessTests (202.97s)
Minitest::UnexpectedError:         NameError: undefined local variable or method `categories' for an instance of ProgrammingEnvironment
            app/models/programming_environment.rb:146:in `block in categories_for_navigation'
            app/models/programming_environment.rb:145:in `categories_for_navigation'
            app/controllers/programming_classes_controller.rb:89:in `show_by_keys'
            app/controllers/application_controller.rb:187:in `with_locale'
            test/test_helper.rb:506:in `block (2 levels) in test_user_gets_response_for'
            test/testing/capture_queries.rb:27:in `assert_queries'
            test/test_helper.rb:505:in `block in test_user_gets_response_for'
            test/testing/setup_all_and_teardown_all.rb:36:in `run'

It will only work for attributes, this was deprecated in Rails 7.1, but actually dropped in Rails 7.2. We aren't the only ones feeling some pain from this, see stackoverflow: aliasing associations in rails 7.1 for example. Shopify opened a PR that added an alias_association feature, which is still open, but did not get merged into Rails 7.2 unfortunately: rails/rails#49801

Workarounds

  • Use alias_method. This should work for read-only associations, but will not work for property= association methods.
  • A related rails issue (has_many through alias_method rails/rails#51062) suggests a workaround of adding multiple associations that point to the same table.
  • Avoid using the alias: in some cases we may only use one or the other other of the aliased attribute. In this case, removing the unused association is an obvious fix that simplifies code.

Fixing

A quick search suggests we use alias_attribute in 10 places.

5 of these places its pretty clearly an actual attribute being aliased, and I think nothing needs changing:

dashboard/app/models/backpack.rb:
  20    # to reflect the new table name, so an alias is used to clarify which table this ID maps to.
  21:   alias_attribute :project_id, :storage_app_id
dashboard/app/models/channel_token.rb:
  30:   alias_attribute :project_id, :storage_app_id
dashboard/app/models/featured_project.rb:
  22:   alias_attribute :project_id, :storage_app_id
dashboard/app/models/project_commit.rb:
  20:   alias_attribute :project_id, :storage_app_id
dashboard/app/models/course_version.rb:
  55:   alias_attribute :version_year, :key

For this one, I dropped :programming_environment_categories, which was afaict unused in favor of its alias categories:

dashboard/app/models/programming_environment.rb:
  26:   alias_attribute :categories, :programming_environment_categories

After change, related tests still pass: bundle exec spring testunit ./test/models/programming_environment_test.rb

Fixed by: 4c94df7


As far as I can tell RegionalPartnerCohort is actually an unused vestigial table. I'm following up with acquisition products to confirm this. Nevertheless, I ported it, following the "duplicate the association" pattern, in case there's some stealth use of it I can't confirm this is the safest pattern

dashboard/app/models/pd/regional_partner_cohort.rb:
  39:   alias_attribute :members, :users

After change related tests pass: bundle exec spring testunit ./test/models/pd/regional_partner_cohort_test.rb

Fixed by: 3486912

Update: @dmcavoy confirms that this is probably a vestigial table that we can (should!) drop


Here I discovered that two of the aliases were real attributes. I wasn't sure what qualified, with grades having a serializer, but I tested it and it works fine. For alias_attribute :teacher, :user I wasn't able to convince myself that one or the other could be reliably switched to because the words are generic and hard to search for. Thus I followed the recommended pattern of just duplicating the association.

dashboard/app/models/sections/section.rb:
  66:   alias_attribute :teacher, :user 
  92:   alias_attribute :lesson_extras, :stage_extras # this looks like a real attribute
  98:   alias_attribute :grades, :grade # I tested this and its a real attribute

Fixed by: 5f4f7fd

Base automatically changed from seth/ruby-3.3 to staging January 9, 2025 03:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant