Rails 404, 406 and the X-Cascade hack: how a dot in a URL picks your status code

Last week I tweeted that asking Rails for .php in a URL gives you 406, not 404, and that my colleague assumed this was the HTTP standard. It is not. It is a Rails decision from 2006, and it sits on top of three separate mechanisms that decide what "not found" means.

Rails is proud of convention over configuration, yet showing one nice 404 page takes all three of them. They were added years apart, and they talk to each other through an HTTP header and a path rewrite.

The tweet mentioned Rails 8.1.4. The main experiment below ran on Rails 6.1.7.6, the app I was debugging. I reran the format checks on a clean Rails 8.1.4 app with Rack 3.2.8. Those results have their own section, and one of them changes the X-Cascade advice later in the post.

What HTTP actually says

HTTP has no concept of a file extension. A dot in the path means nothing, and /report.php is just a string that identifies a resource.

The format of a response is chosen by content negotiation, mostly the Accept header. The two status codes mean different things:

• 404: the server has no current representation of this resource, or will not admit to having one.
• 406: the resource exists, but no representation fits what the client asked for in its negotiation headers.

So a 406 for /wp-login.php, sent by a client with Accept: text/html, does not fit the definition. The client asked for HTML, and nobody negotiated anything.

The Rails side of this story starts with commit 7611fa6d46 by David Heinemeier Hansson, December 2006:

Added Request#format to return the format used for the request as a mime type. If no format is specified, the first Request#accepts type is used.

The whole feature was one line:

parameters[:format] ? Mime::Type.lookup_by_extension(parameters[:format]) : accepts.first

From then on, the extension in the URL outranks the header. This was a good idea for 2006, when browsers sent useless Accept values and /posts/5.xml was a pleasant way to ask for XML. The 406 came on top of it later: today ActionDispatch::ExceptionWrapper.rescue_responses maps ActionController::UnknownFormat to :not_acceptable, so a URL suffix can end in a status code that is defined in terms of headers.

The experiment

A single file is enough to see all of this. You need gem install rails -v 6.1.7.6, then ruby app.rb:

gem 'rails', '6.1.7.6'
require 'action_controller/railtie'
require 'rack/mock'

class MissingRecord < StandardError; end

class DemoApp < Rails::Application
  config.eager_load = false
  config.hosts.clear
  config.secret_key_base = 'x'
  config.logger = Logger.new(nil)
  config.consider_all_requests_local = false
  config.action_dispatch.show_exceptions = true
  config.action_dispatch.rescue_responses['MissingRecord'] = :not_found

  routes.append do
    get '/things/:id', to: 'things#show'
    get '/strict/:id', to: 'things#show', id: /[^\/]+?/, format: /[a-z]+/
    get '/htmlonly/:id', to: 'things#html_only'
  end
end

class ThingsController < ActionController::Base
  def show
    raise MissingRecord unless params[:id].start_with?('1')
    render plain: "id=#{params[:id].inspect} format=#{params[:format].inspect}"
  end

  def html_only
    respond_to { |f| f.html { render plain: 'html page' } }
  end
end

DemoApp.initialize!

%w[/things/1 /things/1.37054 /things/1.php /things/6.37054 /htmlonly/1 /htmlonly/1.json /htmlonly/1.php
   /strict/1.37054 /strict/6.37054 /nothing/here].each do |path|
  status, _, body = DemoApp.call(Rack::MockRequest.env_for(path, 'HTTP_ACCEPT' => 'text/html'))
  text = +''
  body.each { |chunk| text << chunk }
  puts "#{path.ljust(20)} #{status} #{text}"
end

p DemoApp.routes.call(Rack::MockRequest.env_for('/nothing/here')).first(2)

Note the two rows for /things. An article number like 6.37054 is not rejected by the default route. It is cut in half, and your action happily looks up record 6. If that record exists, the user gets the wrong page with a 200.

I also tried an action with no respond_to at all, only a show.html.erb, because .php is less consistent than my tweet suggested:

So the same URL shape gives 200, 406 or 500 depending on how the action is written, and none of the three is 404.

And the empty body for .php? PublicExceptions looks at the request format, finds nothing it can serialize, tries public/406.html, and Rails does not generate that file. Then it returns X-Cascade: pass, and that header is the next topic.

The same checks on Rails 8.1.4

I generated a clean Rails 8.1.4 app (Rack 3.2.8, Ruby 3.2.3) with five routes and four controllers, and requested every URL through ActionDispatch::Integration::Session:

• implicit: def show; end with only a show.html.erb
• explicit: respond_to { |format| format.html }
• legacy: class-level respond_to :html, :js, :json and respond_with from the responders gem
• lowercase: the implicit controller on a route with id: /[^\/]+?/, format: /[a-z]+/
• strict: id: /[^\/]+?/, format: false, and the action raises ActionController::RoutingError for unknown ids

So the tweet holds on 8.1.4, with a footnote: .php gives 406 as soon as the action declares its formats (respond_to or respond_with), and 200 when the action renders implicitly. Nothing here differs from 6.1.

Three mechanisms for one "not found"

The router. When nothing matches, ActionDispatch::Journey::Router#serve ends with this line and raises nothing:

[404, { "X-Cascade" => "pass" }, ["Not Found"]]

Exceptions. A controller raises ActiveRecord::RecordNotFound or something else. ActionDispatch::ShowExceptions rescues it, asks ExceptionWrapper for the status, rewrites PATH_INFO to "/404", changes the request method to GET, and calls the exceptions app with the modified env. There is a piece of glue between the first two mechanisms: ShowExceptions only understands exceptions, so DebugExceptions checks every response for X-Cascade: pass and turns it into ActionController::RoutingError. That is the "No route matches" you know from your logs. It is produced by a middleware reading a header, not by the router.

config.exceptions_app. The default is ActionDispatch::PublicExceptions.new(Rails.public_path). It serves public/404.html and public/500.html for HTML, and a small body for formats it can serialize, such as JSON.

Together, with a custom exceptions app, it looks like this:

sequenceDiagram
    autonumber
    participant C as Client
    participant S as ShowExceptions
    participant D as DebugExceptions
    participant R as Router
    participant A as Controller
    participant E as exceptions_app
    participant X as ErrorsController

    C->>S: GET /products/1/nope
    S->>D: call(env)
    D->>R: call(env)
    alt no route matches
        R-->>D: 404 + X-Cascade: pass
        Note over D: pass becomes RoutingError
        D--xS: raise RoutingError
    else route matches
        R->>A: dispatch
        A--xS: raise RecordNotFound
    end
    Note over S: status from ExceptionWrapper<br/>PATH_INFO = /404, method = GET
    S->>E: call(env)
    E->>R: routes.call(env)
    alt status is routed
        R->>X: dispatch to not_found
        X-->>E: 404 + HTML page
    else status is not routed, like /500
        R-->>E: 404 + X-Cascade: pass
        Note over E: fallback to PublicExceptions
    end
    E-->>S: response
    S-->>C: response

Yes, the router gets called twice. The second time, your error page is a normal route that happens to be named after a status code.

In development config.consider_all_requests_local = true makes DebugExceptions render its debug page, so the request never reaches exceptions_app. You will not see your 404 page locally unless you flip that flag.

The older, leakier way: rescue_from

Before exceptions_app was common, apps caught the error in the controller. Spree had this helper for years:

rescue_from ActiveRecord::RecordNotFound, with: :render_404

def render_404
  respond_to do |type|
    type.html { render status: :not_found, file: "#{Rails.root}/public/404.html", layout: nil }
    type.all  { render status: :not_found, nothing: true }
  end
end

render nothing: true was removed in Rails 5.1. Removed does not mean rejected. On Rails 6.1 the option is silently ignored, and render falls back to its default: the action's own template. That template expects the record that was just not found.

sequenceDiagram
    autonumber
    participant C as Client
    participant R as Router
    participant A as OrdersController
    participant H as render_404
    participant T as show.json template
    participant S as ShowExceptions

    C->>R: GET /orders/1.json
    R->>A: dispatch to show
    A--xH: rescue_from RecordNotFound
    H->>H: respond_to picks type.all
    Note over H: nothing: true is ignored
    H->>T: default render
    T--xS: NoMethodError on nil record
    S-->>C: 500

I reproduced this on 6.1.7.6: a NoMethodError for number on nil, status 500, for .json and for .php. The HTML branch keeps working, so nobody notices until you read the error tracker. The handler for "not found" crashes while handling "not found".

Upstream had already moved on. Spree first switched this branch to head :not_found, deprecated the helper, and removed it in commit 704baa4, May 2018, which shipped in 3.7. Solidus removed it in PR #2329, October 2017, shipped in 2.5, with the best one-line summary of this whole post:

Rails knows how to render a 404, so this is unnecessary and can hide errors.

The trap survives in long-lived forks, where a copy of the old helper outlives the option it depends on.

What the router does with a dot

The routing guide warns about this:

By default, dynamic segments don't accept dots - this is because the dot is used as a separator for formatted routes.

It then suggests the constraint id: /[^\/]+/. That fixes the id, but this greedy regex also eats the format, so /greedy/1.json gives id="1.json". If you still want .json to work, you need a lazy id and a format constraint together. Neither is enough alone: with only the lazy id, 6.37054 is split again.

flowchart TD
    U["GET /things/6.37054"] --> Q{"Route definition"}
    Q -->|"default"| A["id = 6<br/>format = 37054"]
    A --> A2["Wrong record or 404"]
    Q -->|"id: greedy regex"| B["id = 6.37054<br/>format = nil"]
    B --> B2["Works, but .json is eaten too"]
    Q -->|"id: lazy regex<br/>format: a-z only"| C{"Is the last suffix<br/>lowercase letters?"}
    C -->|"yes, .php or .json"| D["id = name before the dot<br/>format = suffix"]
    C -->|"no, 37054 or B"| E["id = whole string<br/>format = nil"]

Two things surprised me here. First, format: /[a-z]+/ does not make /strict/1.37054 a routing miss. I expected that at first, but the lazy id simply keeps growing until the whole string matches, and the optional format stays empty. Second, the constraint does not get rid of .php, which is still a valid lowercase format. /strict/wp-login.php gives id="wp-login" and format="php". And an id with a lowercase suffix, like 000302.b, would still be split. The constraint fits identifiers that end in digits or capitals, not arbitrary strings.

If your URLs never need a format, format: false from the format segments section is simpler. With the default id, /nof/1.php becomes a clean routing 404, because nothing is left to absorb the suffix. That is the only option here that really makes .php a 404 at the router.

The X-Cascade hack

The common advice for a nice error page is config.exceptions_app = routes with a route for /404. This is a community recipe, and I could not find it in the guides, which only mention the default value. It works for 404, but it has a quiet failure: for a status you did not route, routes.call(env) answers 404 with X-Cascade: pass, ShowExceptions replaces that with an empty response, and your public/500.html is gone. On my test app /boom went from the static page to a 500 with an empty body.

So the application needs a fallback:

config.exceptions_app = lambda do |env|
  response = routes.call(env)
  response[1]['X-Cascade'] == 'pass' ? ActionDispatch::PublicExceptions.new(Rails.public_path).call(env) : response
end

This is the line that bothers me. It mirrors show_exceptions.rb:54 almost exactly:

response[1]["X-Cascade"] == "pass" ? pass_response(status) : response

On Rails 8.1.4 the same line reads response[1][Constants::X_CASCADE] == "pass". That difference matters, as you will see below.

Where does this header come from? The idea is older than Rails error pages: "this app declines, ask the next one". Rack::Cascade has the same idea, trying apps in order until one answers. The funny part is that Rack::Cascade does not read X-Cascade at all, it only looks at the status (404 and 405 by default). The header is a convention between routers: Rails uses it, and Sinatra sets it too for unmatched requests.

The glue was written by José Valim in one day, 2011-12-16. Commit deef8dd68 extracted PublicExceptions into a Rack app, which made error pages replaceable. Commit 535926269 taught ShowExceptions to understand a "pass" from that app, and PublicExceptions started returning it for a missing file. That same commit deleted a comment above the check in DebugExceptions:

# TODO: Maybe this should be in the router itself

The comment went away. The check stayed where it was, and on 6.1 it is still there. ShowExceptions itself dates from 2009, by Joshua Peek.

Each step was reasonable. The sum is that a Rack-level signal between middlewares leaks into config/application.rb, and a framework that generates a whole CRUD from one command asks you to index into a response tuple by hand. There is even a practical cost, and this time I measured it. Rack 3 requires lowercase header names. On Rails 8.1.4 with Rack 3.2.8 the router answers 404 with the header x-cascade, ActionDispatch::Constants::X_CASCADE is "x-cascade", and response[1]['X-Cascade'] returns nil. The lambda above still runs, but the comparison is never true, so it returns the router's bare 404 instead of PublicExceptions, and public/500.html quietly disappears after the upgrade. Use ActionDispatch::Constants::X_CASCADE. Rails 8.1.4 has it, 6.1.7.6 does not, and I did not check which release introduced it.

A nicer API would be small: config.exceptions_app = routes that falls back to public/*.html per status by itself. The upstream issue tracker shows that this corner is not comfortable: cookies in 404 pages, invalid MIME types with exceptions_app = self.routes, UnknownHttpMethod.

The recipe

This is what I ended up with. I ran the recipe on 6.1.7.6, and I checked the header and the format behavior on 8.1.4.

1. Stop mis-parsing identifiers. Only lowercase letters count as a format:

   get '/products/:product_id/:id', to: 'variants#show', id: /[^\/]+?/, format: /[a-z]+/
   

2. Look the record up by canonical slug or normalized id (6.37054 to 6-37054), and answer with a 301 to the canonical URL so the page is not indexed twice.

3. Delete render_404 and every rescue_from for not-found errors.

4. Render every 404 in one place, keeping PublicExceptions for the rest with the lambda above (compare against ActionDispatch::Constants::X_CASCADE instead of the string on Rails 8):

   match '/404', to: 'errors#not_found', via: :all
   

   class ErrorsController < ActionController::Base
     def not_found
       return head :not_found if request.format.json? || request.format.js?
       render formats: :html, status: :not_found
     end
   end
   

   The formats: :html is what lets .xml or .webmanifest get the normal page. request.format still reflects the original request, and request.original_fullpath still has the original URL. You may see advice to skip forgery protection here because the original request can be a POST. On 6.1.7.6 I did not need it: ShowExceptions rewrites the method to GET, and a POST with no token got the normal 404 page.

5. Map stray missing templates to 404:

   config.action_dispatch.rescue_responses['ActionView::MissingTemplate'] = :not_found
   

   It trades a 500 for a 404, so a template that is really missing will no longer page anyone. Decide if you can live with that.

6. Decide what .php on an existing record should mean. With the steps above, scanner traffic like wp-login.php gets a normal 404, because the lookup fails before respond_to runs. But /products/1/6.37054.php for a real record is still 406 with an empty body. I tried a before_action that answers early, but it also changed the .json output of single records while list endpoints kept theirs, so I dropped it. The one-line alternative is this:

   config.action_dispatch.rescue_responses['ActionController::UnknownFormat'] = :not_found
   

   It works, but then a real Accept: application/json on an HTML-only page also gets 404, and that was the one case where 406 was correct. I left the default. Scanners rarely guess a real article number.

So is it the HTTP standard?

No. HTTP says nothing about extensions, and its 406 is about negotiation headers. Rails treats a URL suffix as a negotiation result, because in 2006 that was useful. The convention still works well for the case it was built for, which is .json and .xml. It only gets strange when a dot means something else, like an article number or a scanner probing for PHP.

What I took from this: read the status code as a trace of which road the request took. A 404 can be a route miss, a raised exception or a failed lookup behind a cut id. A 406 is almost always a format the action cannot serve. A 500 on a missing record means someone is still handling errors by hand.

References

• Rails Routing guide: Dynamic Segments, Format Segments
• Rails Configuring guide, config.exceptions_app
• API: ActionController::MimeResponds, ActionDispatch::PublicExceptions, ActionDispatch::ShowExceptions
• Rack::Cascade
• RFC 9110, HTTP Semantics
• Rails commits: 7611fa6d46 (Request#format, 2006), 11af089ce (ShowExceptions, 2009), deef8dd68 and 535926269 (exceptions app and X-Cascade, 2011)
• Spree commit 704baa4 and Solidus PR #2329
• The tweet that started this