Given that this is clearly a lot of work, what's the motivation for this section/this new set of docs? What problem/questions from folks are you seeing that you're trying to solve here that's not currently being addressed in the gallery?

I'll ping for a review for this. Obviously this can just be considered scaffolding.

I believe I addressed why the gallery is not a replacement for explaining how to make plots in the User Guide in the PR description. The gallery will never be a cohesive introductory guide, and I think that is fine. The approach here is to provide a basic start with links to more examples in the appropriate section of the gallery, or elsewhere in the User Guide.

I'm slightly annoyed by the structural duplication of this page and "Plot types".

This was deliberately done so that they would present material in the same order but with more depth than in the quick visual index of "Plot Types". I'm not sure I understand the annoyance - particularly given that I made the Plot Types PR.

MHO this would work better if

you first create an issue describing the topic, and the idea for the stuctural solution (optionally/alternatively make a proof-> of-concept PR for the structure). We could then agree on a the structure first.
then make a PR that puts the desired structure in place
after that, add multiple PRs for the individual sections.
This would keep the PRs smaller and more focused and significantly ease reviewing.

I'd consider this the proof of concept PR. If you have serious objections to the structure, folks could propose alternates. If not, I'd suggest this PR is better than nothing, which is what we currently have. If folks disagree and feel it is worse than nothing, then we should close the PR and continue to have a large gap in the docs.

If folks disagree and feel it is worse than nothing, then we should close the PR and continue to have a large gap in the docs.

I'm not sure what gap it is you're trying to fill with these docs. The plot type gallery is already heavily curated, so I don't think that the narrative is adding much here.

Some of the examples feel so bogged down with extras that I'm not sure that readers will pick up the pattern that I think is what you're trying to convey, and information on how to use each specific function.

I think if there's a gap here, it's in understanding the artists that back the plots and I'd orient the user guide plotting section around those underlying artists.

The User guide has how to make a figure, how to make an axes, ...., a bunch of complicated stuff. I don't think "go look at the plot types gallery" is a good substitute for that middle ground, and I don't think the plot types gallery ties visualizations together in a hierarchy other than their ordering. Which is great - I look at the plot type gallery as a quick overview of what the library can do that a user can quickly scan to see if their chosen visualization is there, and see what we call it.

This new section of the User Guide should get into more details and connect the visualizations which is what I tried to start here. For sure, adding more detail about the underlying artists could be helpful, though I'd perhaps not ruin the flow so much as show a few examples of using the artists, and link out to the API docs or explanation elsewhere in the the Guide.

Some of the examples feel so bogged down with extras

I'm not sure what you are referring to here, but the examples are meant to be semi complete visualizations in most cases. If folks feel something got too far into the weeds, they are welcome to propose scaling a section back or breaking it into components.

I do somewhat feel you are expressing two contradictory ideas of what this section is meant for; "more details about underlying artists" and "fewer extras" are pulling in opposite directions. Obviously writing examples is a balancing act, and folks should feel free to edit this first pass all they want.

The User guide has how to make a figure, how to make an axes, ...., a bunch of complicated stuff.

That's because the user guide introduces the concept of a figure, axes, colormap, etc, all of which are backed by objects. There's not a concept of a plot in the same manner, the closest being the artists that underlie the plot. And it covers the basics of how to make a plot in getting started and in the discussions of the library components relevant for that plot.

I'm not sure what you are referring to here, but the examples are meant to be semi complete visualizations in most cases.

For example, this gets so bogged down in the user warning (which is better explained in the API docs) that it loses focus on how to use the function (if that's the goal)
https://github.com/matplotlib/matplotlib/blob/2a0455e40a990bec76f671ef2d9be0cf7f6ea4af/galleries/users_explain/plotting/gridded.py#L95C5-L118

For example, the focus here is on setting up the data, the actual usage of the function not changing from the usage above:

For example, the goal of this example is presumably to contrast the two, which a reader can pick up from the gallery thumbnails:

https://github.com/matplotlib/matplotlib/blob/2a0455e40a990bec76f671ef2d9be0cf7f6ea4af/galleries/users_explain/plotting/gridded.py#L150C5-L173

I do somewhat feel you are expressing two contradictory ideas of what this section is meant for; "more details about underlying artists" and "fewer extras" are pulling in opposite directions.

What I mean by focusing on the artist is more documents in the style of https://matplotlib.org/devdocs/users/explain/artists/imshow_extent.html. It unpacks a characteristic of ImageBase objects that once the reader understands it, they can then apply that understanding to every plotting method that takes an extent keyword.

That's because the user guide introduces the concept of a figure, axes, colormap, etc, all of which are backed by objects. There's not a concept of a plot in the same manner, the closest being the artists that underlie the plot. And it covers the basics of how to make a plot in getting started and in the discussions of the library components relevant for that plot.

From my point of view people need to be able to make a figure, make and axes, and then make visualizations in those figures and axes. The User Guide as it stands does not have that last step. They are all underscored by objects, so I'm not following what that has to do with explaining how to use the library.

For example, this gets so bogged down in the user warning (which is better explained in the API docs) that it loses focus on how to use the function (if that's the goal)

What I mean by focusing on the artist is more documents in the style of https://matplotlib.org/devdocs/users/explain/artists/imshow_extent.html.

I'd say both of these are somewhat asides that help the user better use the library. You could easily argue that the asides should be in a separate page or put in-line. But I don't think a deep-dive on every artist type and its quirks is the right level for the top-section of the user guide. From my point of view, if it can be explained quickly, inline is OK, if not then it should be linked out to a dedicated subpage.

I give one high-level comment, other than that I'm out of the details here, because the PR gives me a mental overload:

I definitively see a place for a more detailed usage description of (some) plotting functions.
Examples:

• plot() needs explanition at least on (i) plotting lines and/or markers, (ii) the use of the shorthand fmt string plot(x, y, 'r-') vs. keyword arguments; (iii) an explantion that and why the result is a list of Line2D and not a single Line2D plus the tuple-unpacking trick to put it into a single variable.
• imshow() should have a basic explanation how to use color limits and colormaps for simple cases (not going deep dive into the data->color pipeline).

The user guide is certianly the right place for this. (How to plot with lines / markers / images). I'm not clear though on the framing / perspective. I'm feeling that the focus on "plotting methods" (Document: Using matplotlib > Plotting methods, and using method names as the sections) is wrong for a user guide. I think it should instead be something like "Using matplotlib > Visualizing different kinds of data". Then the subsections "pairwise data", "statistical plots", ... in here are ok. But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.

From my point of view people need to be able to make a figure, make and axes, and then make visualizations in those figures and axes. The User Guide as it stands does not have that last step. They are all underscored by objects, so I'm not following what that has to do with explaining how to use the library.

Because those sections are focused on explaining what a figure (object) is and how to create it, what an axes (object) is and how to create it, etc.

@timhoffm is also suggesting by artist here, just implicitly:

But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.

Because each of those visual elements is backed by a different artist.

@timhoffm is also suggesting by artist here, just implicitly:

But within these sections again, it should not be methods but "drawing lines, markers, bars, ..." etc.
Because each of those visual elements is backed by a different artist.

To be precise, I'm suggesting by kind of visualization. There's a strong parallel between "kind of visualization" and Artists, but it's not exactly 1:1 (lines = Line2D, markers = Line2D). And my point is that the user guide should be written from the perspective what the user wants (e.g. a color-coded visualization of 2D data) not what we provide (a method imshow() or an AxesImage). We should structure by the former and explain how to realize it using the latter.

And my point is that the user guide should be written from the perspective what the user wants

Sure agreed. That is basically what the current version does. It explains the type of data to be visualized and then explains the different methods to accomplish that. We don't want these to be completely abstract. Users should learn out names for things. And they should also learn the differences between the methods.

Because those sections are focused on explaining what a figure (object) is and how to create it, what an axes (object) is and how to create it, etc.

I think you are making a distinction here that the original author of those sections didn't intend.

However, the practical difference between an Axes object and a Line2d object is that you need to call many methods on the Axes object to use the library at all (or use pyplot). On the other hand, you could make hundreds of complicated visualizations using Line2d artists and not know any of the methods on that artist, or that the artist object even exists.

Not at all to argue that these sections should avoid mentioning or using artists, just that I don't think that should be a focus at the top level.

That is basically what the current version does. It explains the type of data to be visualized and then explains the different methods to accomplish that.

But what @timhoffm (and I agree) is saying is it should be grouped by visualization task. I think some of this is fixable with subheadings saying things like "Plotting lines", etc...

I think what's getting jumbled up here is that method and task aren't a 1:1 either. Like scatter plots can be made using .plot or .scatter, line plots can be made using .plot, and for special types of lines .vlines, .hlines, .axline, .event and
.stem, .stair and .step (all of which I think can be accomplished with .plot)

But what @timhoffm (and I agree) is saying is it should be grouped by visualization task. I think some of this is fixable with subheadings saying things like "Plotting lines", etc...

The current headings are:

• Plotting Methods
  • Pairwise data
  • Statistical plots
  • Gridden plots
  • Unstructured data
  • 3d and volumetric data

The sections start with a short intro about the type of data to be visualized, and then are indeed subsectioned by method type. So taking "Pairwise data" , it has

• plot
• fill_between, fill_betweenx, and stackplot
• errorbar
• scatter
• bar and stem
• step and stairs

we could maybe expand the description of things you do with pairwise data a bit at the beginning, but this is pretty obvious thing to do with your data. We could also change the headings, but to what without being too verbose? "plot" -> "plotting lines and markers, one color and marker at a time" "fill_between"-> "indicating regions around a line"?

The first sentence of each of these sections basically says what the primary goal of the method is, so in my opinion it's clearest just to use the method name. New users may not know our names for things, but the hope is they can realize they want to plot pairwise data, and then skim the plots for the method most suited to their visualization and then read more.

Like you said, the goal here is big picture teach about the plotting API. Not every single plotting function, just why they work the way the do.

I don't thinks I said that. I don't consider this a "why" document at all any more than the rest of the Users Guide. It's meant to be a "How" document.

I'm pretty loathe to go back to the drawing board, and frankly do not have time for it. My apologies that this is not very collaborative, but not everything is best done by committee. Again, the project should feel free to accept these changes and then improve over time, or reject, and leave the docs with this hole in it.

https://diataxis.fr/explanation/ has a lot more detail than the table.

A key line from that page

It’s documentation that it makes sense to read while away from the product itself (one could say, explanation is the only kind of documentation that it might make sense to read in the bath).

This is a coherent prose document that if we pointed someone with a reasonable scientific training at they could read start-to-finish, never go run a line of code, and come out with a solid basis to start using Matplotlib effectively. This is exactly the docs I wish I could point many of the people I support at my day-job at.

While it is true that this do not go into the history of every design decision, it does help the user decide which of several similar looking tools (imshow, pcolor, pcolormesh, ...) to reach for or provide some justification of why the kwrag on hist is density. This is a good start and can be built on to add more of that context (or more links to other pages to keep these readable) in more review able chunks.

I think the "Why...?" this is answering is "why does this even exist? What unique visualization problem is it solving that is not already solved by some other function?"

I am very much against any proposal to cut this down, the core value is the complete narrative. Again quoting diataxis:

When writing explanation you are helping to weave a web of understanding for your readers. Make connections to other things, even to things outside the immediate topic, if that helps.

To the organization concern, I think that this is the right skeleton to start reorganizing every other document in user/explain onto. Currently it is a scattering of correct but unconnected documents. Once this is merged, for every other document we can ask "should this be merged into the narrative or linked to as an appendix (or split in half between the two)?" and then start (slowly) pushing these documents into something coherent. That may be a bigger goal than @jklymak had in mind, but on thinking about it that is the obvious organizing principle that we are currently lacking!

then start (slowly) pushing these documents into something coherent.

That is the goal - once the basic outline is there, I think it's relatively straight forward to iterate and make things more cohesive.

I do have concerns on structure, content, style and quality. As originally said, I feel unable to properly review and address them due to the size of the PR. Therefore I'll recurse myself from this discussion and let others decide how to proceed.

My only request would be that you add a documentation section next to https://matplotlib.org/devdocs/devel/document.html#examples-guidelines and https://matplotlib.org/devdocs/devel/document.html#plot-types-guidelines that describes intent and recommendations of this new part of the docs. Otherwise I'm afraid everybody will have their own mental model how this should look like and future iterations will not converge to a consistent or desired state.

but not everything is best done by committee

That's basically how open source works.

Again, the project should feel free to accept these changes and then improve over time, or reject, and leave the docs with this hole in it.

My concern is the size of this PR will make it really difficult/overwhelming to go in and make changes, so effectively we're stuck with it. I don't think this hole is so urgent to fix that we need to rush a solution.

And come out with a solid basis to start using Matplotlib effectively.

They could do that from the quickstart guide. There's really nothing extra here from a conceptual standpoint that they can't find in our API docs or examples.

This is exactly the docs I wish I could point many of the people I support at my day-job at.

Can you find someone and point them at this PR and see if it works for them? Built pages B/c frankly the more times I read this PR, the more I see small details that in my experience would derail a learner.

ETA: what I mean is there are a lot of descriptive notes - use this aspect ratio, the function does this - but it's missing the why the reader needs to know this/why is the tutorial telling the reader this.

why does this even exist? What unique visualization problem is it solving that is not already solved by some other function?

Almost all that "why" is already in the plot gallery preamble. I think the user guide should explain what the user needs to know about why Matplotlib solves this visualization problem this way so that they can use the functions effectively.

then start (slowly) pushing these documents into something coherent

Who is doing that work w/o consensus/buy-in that this is what we want?

Once this is merged, for every other document we can ask "should this be merged into the narrative or linked to as an appendix (or split in half between the two)?" [...] But on thinking about it that is the obvious organizing principle that we are currently lacking!

I don't follow what you mean here and therefore am not sure how this makes any sense for the structural artist docs (figure/axes/etc) which are currently discoverable. Which currently the implicit organizing principle of the user guide is the "anatomy of Matplotlib" figure, which I think makes sense if the goal is learning Matplotlib concepts.

My only request would be that you add a documentation section

I think that should be a separate PR that comes first b/c that discussion would get to the consensus that I think this PR needs.

My only request would be that you add a documentation section next to https://matplotlib.org/devdocs/devel/document.html#examples-guidelines and https://matplotlib.org/devdocs/devel/document.html#plot-types-guidelines that describes intent and recommendations of this new part of the docs.

Sure, done. I added a section on tutorials as well since that was missing, and updated the table to have a bit of info about the goal of each section of the galleries, made that a subsection, and added a link near the table to the subsection.

Note this is a separate commit: 2bb91bb

Edit: see https://output.circle-artifacts.com/output/job/f5d8cef0-00ee-4839-9149-67e13aa61731/artifacts/0/doc/build/html/devel/document.html#create-examples-and-tutorials

On a side note, I believe our "User Guide" is both "Explanation" (e.g. Introduction to Artists) and "How-to" (e.g. Faster Rendering Using Blitting). That may or may not be good, but is out of scope for this PR.

The important part is that we agree on which category "Plotting data" is. I agree with @jklymak (#29124 (comment)) that "Plotting data" is a "How-to". Essentially,

1. How do I plot [data type]"? - Decide on the kind of visualization you want and follow up with the respective plotting function in 2.
2. How do I use [plotting function]? - Learn how to pass your data into the function and some relevant customizations (mostly kwargs).

I like the tutorials section. 👍

I'm unclear on the user guide section, which also stems from the fact that the user guide is currently "How-to" and "Explanation". I would not bother on trying to define this now and here. Just the "plotting data" section should be enough.

I'm unclear on the user guide section, which also stems from the fact that the user guide is currently "How-to" and "Explanation". I would not bother on trying to define this now and here. Just the "plotting data" section should be enough.

Sorry to cross talk in #31496...

As I state there, I think it's awkward to discuss a subsection of the User Guide like it is standalone. The point of this PR is that it is a gap in the current User Guide.

The important part is that we agree on which category "Plotting data" is. I agree with @jklymak (#29124 (comment)) that "Plotting data" is a "How-to". Essentially,

How do I plot [data type]"? - Decide on the kind of visualization you want and follow up with the respective plotting function in 2.
How do I use [plotting function]? - Learn how to pass your data into the function and some relevant customizations (mostly kwargs).

The plot type gallery solves one, the links out into the API docs which point to the minigallery that goes to the examples solves two. So what's the value add here? ETA2; What I mean is, what makes this different enough from the minigallery->examples pathway that it's worth maintaining a whole set of docs rather than signposting that pathway better? (We can always add a "what's next? learn to plot through the example gallery" card)

ETA: in particular I'm thinking of examples like pie.

So what's the value add here? ETA2; What I mean is, what makes this different enough from the minigallery->examples pathway that it's worth maintaining a whole set of docs rather than signposting that pathway better? (We can always add a "what's next? learn to plot through the example gallery" card)

ETA: in particular I'm thinking of examples like pie.

Examples are rather anecdotal. They teach by demonstration, rather than by explanation or instruction. Because the visualization and preview image have a strong weight, I'd consider it rather an anti-pattern for examples to have many plots.

In that sense, some of the examples, including pie are actually very much the type of content, I'd like to see under "Plotting data" and should be moved there. (they are currently in the examples category because we don't have a dedicated place for them).

Another example of "Plotting data" content: plot() should roughly describe accepted data (plot(y), plot(x, y), plot(x, Y)), styling using kwargs, styling shortcuts using fmt. That's not something we can reasonably put into an example.

some of the examples, including pie are actually very much the type of content, I'd like to see under "Plotting data" and should be moved there

Agree, argued something similar in #27459 or the discussions around it, (which honestly my motivation there is the same as one of my concerns here - focus is too much on what's being plotted, not enough on how to plot it) and discussions around it. But that's also a very different approach from the one taken in this PR. I'm also deeply skeptical of the "well then rework on next PR to that style" b/c it will run headlong into the same issues as #27459

some of the examples, including pie are actually very much the type of content, I'd like to see under "Plotting data" and should be moved there.

Yes, that'd be great - not including pie is an oversight, not a deliberate omission. There are likely others.

I'd indeed go through the Examples and do some pruning - some of the material is already cribbed from examples, and probably more could be done.

As a side note/feedback: I wasn't able to easily see whether #29124 (comment) and the subsequent force-push did change pie or whether that was just a communication of intent. This unerpins my point that the PR is too large in scope and content to be reviewable. While I rejoined the conceptual high-level discussion, I still won't systematically look at the actual content of the PR.

No pie is not included. As I'm sure many other things are not either.