Parse jsdoc @param type literals#17352
Conversation
Now Typescript supports the creation of anonymous types using successive
`@param` lines in JSDoc:
```js
/**
* @param {object} o - has a string and a number
* @param {string} o.s - the string
* @param {number} o.n - the number
*/
function f(o) { return o.s.length + o.n; }
```
This is equivalent to the Typescript syntax `{ s: string, n: number }`,
but it allows per-property documentation, even for types that only need
to be used in one place. (`@typedef` can be used for reusable types.)
If the type of the initial `@param` is `{object[]}`, then the resulting
type is an array of the specified anonymous type:
```js
/**
* @param {Object[]} os - has a string and a number
* @param {string} os[].s - the string
* @param {number} os[].n - the number
*/
function f(os) { return os[0].s; }
```
Finally, nested anonymous types can be created by nesting the pattern:
```js
/**
* @param {Object[]} os - has a string and a number
* @param {string} os[].s - the string
* @param {object} os[].nested - it's nested because of the object type
* @param {number} os[].nested.length - it's a number
*/
function f(os) { return os[0].nested.length; }
```
Implementation notes:
1. I refactored JSDocParameterTag and JSDocPropertyTag to
JSDocPropertyLikeTag and modified its parsing to be more succinct. These
changes make the overall change easier to read but are not strictly
required.
2. parseJSDocEntityName accepts postfix[] as in `os[].nested.length`,
but it doesn't check that usages are correct. Such checking would be
easy to add but tedious and low-value.
3. `@typedef` doesn't support nested `@property` tags, but does support
`object[]` types. This is mostly a practical decision, backed up by the
fact that usejsdoc.org doesn't document nested types for `@typedef`.
weswigham
left a comment
There was a problem hiding this comment.
Generally agreeable; there are some refactorings of how jsdoc params and properties are stored which I like. Only thing I'd mention is that there are quite a few ts.isIdentifier which would normally be just isIdentifier (unless there's a reason not to).
The whole name first/name second thing feels a little hairy to deal with; I almost feel it would be neater to handle if each variant were a different SyntaxKind. (I mean, I guess they do parse differently, so it'd make sense) Up to you, though. This feels better than how it was handled before, anyway.
| @@ -1553,6 +1553,10 @@ namespace ts { | |||
|
|
|||
| /** Does the opposite of `getJSDocParameterTags`: given a JSDoc parameter, finds the parameter corresponding to it. */ | |||
| export function getParameterFromJSDoc(node: JSDocParameterTag): ParameterDeclaration | undefined { | |||
There was a problem hiding this comment.
See the (sole) use of this function -- at x in @param obj.x we should return the symbol for x. Add a find-all-references test.
/**
* @param obj.[|x|]
*/
function f(obj) {
obj.[|x|];
}| return declareSymbolAndAddToSymbolTable(node as JSDocPropertyTag, | ||
| (node as JSDocPropertyTag).isBracketed || ((node as JSDocPropertyTag).typeExpression && (node as JSDocPropertyTag).typeExpression.type.kind === SyntaxKind.JSDocOptionalType) ? | ||
| return declareSymbolAndAddToSymbolTable(node as JSDocPropertyLikeTag, | ||
| (node as JSDocPropertyLikeTag).isBracketed || ((node as JSDocPropertyLikeTag).typeExpression && (node as JSDocPropertyLikeTag).typeExpression.type.kind === SyntaxKind.JSDocOptionalType) ? |
There was a problem hiding this comment.
This expression is too large.
const prop = node as JSDocPropertyLikeTag;
const flags = prop.isBracketed || prop.typeExpression && prop.typeExpression.type.kind === SyntaxKind.JSDocOptionalType
? SymbolFlags.Property | SymbolFlags.Optional
: SymbolFlags.Property;
return declareSymbolAndAddToSymbolTable(prop, flags, SymbolFlags.PropertyExcludes);Also, typeExpression is declared non-optional, but here we test for its presence.
| declaration = getDeclarationOfKind<TypeAliasDeclaration>(symbol, SyntaxKind.TypeAliasDeclaration); | ||
| type = getTypeFromTypeNode(declaration.type); | ||
| } | ||
| const declaration = getDeclarationOfKind<JSDocTypedefTag>(symbol, SyntaxKind.JSDocTypedefTag) || |
There was a problem hiding this comment.
getDeclarationOfKind hides a loop, so this loops twice. Could add a version of that function that takes two kinds.
There was a problem hiding this comment.
I'll just use findDeclaration which takes a predicate.
There was a problem hiding this comment.
I was going to delete that function in #17380 since it's equivalent to just find(symbol.declarations, ...) -- this would be one of only 2 uses.
| postParameterName?: Identifier; | ||
| typeExpression: JSDocTypeExpression; | ||
| /** Whether the property name came before the type -- non-standard for JSDoc, but Typescript-like */ | ||
| isParameterNameFirst: boolean; |
There was a problem hiding this comment.
If this applies to more than just parameters, it should be renamed.
| visitNode(cbNode, (<JSDocParameterTag>node).typeExpression) || | ||
| visitNode(cbNode, (<JSDocParameterTag>node).postParameterName); | ||
| case SyntaxKind.JSDocPropertyTag: | ||
| if ((node as JSDocPropertyLikeTag).isParameterNameFirst) { |
There was a problem hiding this comment.
Maybe this should wait for a separate PR, but I would much prefer to see code like:
const { isParameterNameFirst, fullName, typeExpression } = node as JSDocPropertyLikeTag;in all of these cases, instead of repeated casting.
There was a problem hiding this comment.
I will make the change here if you are willing to review it! 😅 I'll put it in a separate commit at the end which might make it easier.
There was a problem hiding this comment.
Please make a separate PR as that would be a huge diff.
| visitNode(cbNode, (<JSDocTypedefTag>node).fullName) || | ||
| visitNode(cbNode, (<JSDocTypedefTag>node).name) || | ||
| visitNode(cbNode, (<JSDocTypedefTag>node).jsDocTypeLiteral); | ||
| if ((node as JSDocTypedefTag).typeExpression && |
There was a problem hiding this comment.
It looks like name is a pointer to a node within fullName. So it would be visited twice if we were recursively calling forEachChild.
There was a problem hiding this comment.
Nice catch! I added fullName later and didn't notice that name needed to be removed.
| } | ||
|
|
||
| const result = shouldParseParamTag ? | ||
| const result: JSDocPropertyLikeTag = target ? |
There was a problem hiding this comment.
Since target is now an enum, use an explicit comparison instead of treating it like a boolean.
|
|
||
| function parseParameterOrPropertyTag(atToken: AtToken, tagName: Identifier, shouldParseParamTag: boolean): JSDocPropertyTag | JSDocParameterTag { | ||
| function isObjectOrObjectArrayTypeReference(node: TypeNode): boolean { | ||
| return node.kind === SyntaxKind.ObjectKeyword || |
There was a problem hiding this comment.
isTypeReferenceNode just tests the syntax kind, so just make that a switch case and have default: return false;.
| skipWhitespace(); | ||
|
|
||
| let preName: Identifier, postName: Identifier; | ||
| let preName: EntityName, postName: EntityName; |
There was a problem hiding this comment.
It no longer makes sense to have separate preName and postName variables. Just create isParameterNameFirst directly.
| } | ||
|
|
||
| function tryParseChildTag(parentTag: JSDocTypeLiteral): boolean { | ||
| function parseChildParameterOrPropertyTag(target: PropertyLikeParse.Property): JSDocTypeTag | JSDocPropertyTag | false; |
There was a problem hiding this comment.
Since there are only 2 possible targets, maybe there should just be 2 different functions parseChildParmeterTag and parseChildPropertyTag that both use parseChildParameterOrPropertyTagCommon?
There was a problem hiding this comment.
I think a more likely refactoring would be to pass 1 of 2 different tryParseChildTag functions instead of the target enum. But the target enum pattern is also used with parseParameterOrPropertyTag, and it's as easy to read and (probably?) as fast to run, so I think I'll keep it this way.
| jsdocTypeLiteral = <JSDocTypeLiteral>createNode(SyntaxKind.JSDocTypeLiteral, start); | ||
| jsdocTypeLiteral.jsDocPropertyTags = [] as MutableNodeArray<JSDocPropertyTag>; | ||
| } | ||
| (jsdocTypeLiteral.jsDocPropertyTags as MutableNodeArray<JSDocPropertyTag>).push(child as JSDocPropertyTag); |
There was a problem hiding this comment.
You could just the create the mutable array locally and assign to jsdocTypeLiteralat the end. That way you avoid casting an already-created object to mutable.
This loop could probably be its own function parseParameterTags returning NodeArray<JSDocPropertyTag> | undefined.
Also, you don't appear to be setting the position on the array -- if you don't want to do that, just declare it as an ReadonlyArray and not a NodeArray.
Also, what's with the cast on child? You passed in PropertyLikeParse.Parameter so it's surprising to see an assertion that what you got back isn't a parameter.
There was a problem hiding this comment.
I don't know a correct position to set on the child array because I don't know what would use it. I don't think I can make it a ReadonlyArray though, because forEachChild expects to iterate over a NodeArray. For now I'll still create a NodeArray with no position.
There was a problem hiding this comment.
You could just use a for loop instead of calling cbNodes, which does require a NodeArray, meaning users might be expecting it to have a position.
| result.postParameterName = postName; | ||
| result.name = postName || preName; | ||
| if (typeExpression) { | ||
| result.type = typeExpression.type; |
There was a problem hiding this comment.
Isn't this redundant? Why do we have both typeExpression and type?
There was a problem hiding this comment.
I wasn't sure how much like a real VariableDeclaration JSDocPropertyLikeTag should become and ended up halfway there. It turns out to be easier to continue special-casing it in getTypeOfVariableOrParameterOrProperty instead of having it become a full-fledged declaration. I dropped type in favour of typeExpression.
| result.type = typeExpression.type; | ||
| } | ||
| result.fullName = postName || preName; | ||
| result.name = ts.isIdentifier(result.fullName) ? result.fullName : result.fullName.right; |
There was a problem hiding this comment.
This is also redundant, I would just make this a function of the node and not a property.
There was a problem hiding this comment.
This was another convenience shim. I turned name into fullName and pushed the isIdentifier check into getNameOfDeclaration.
| result.isParameterNameFirst = postName ? false : !!preName; | ||
| result.isBracketed = isBracketed; | ||
| return finishNode(result); | ||
|
|
There was a problem hiding this comment.
Nit: line before a closing bracket doesn't really accomplish anything -- nothing to separate that isn't already separated by the closing bracket.
| } | ||
| if (child.kind === SyntaxKind.JSDocTypeTag) { | ||
| if (alreadyHasTypeTag) { | ||
| break; |
There was a problem hiding this comment.
What happens to child in this case? We just drop it with no syntax error?
There was a problem hiding this comment.
I think this comment is still unaddressed?
There was a problem hiding this comment.
Yes. It happens in this case:
/**
* @typedef Name
* @type {string}
* @type {Oops} - this is dropped and we stop parsing the @typedef
*/This would be a grammar error if we did more thorough checking of jsdoc structure in the checker (along with a lot of other incorrect-but-allowed jsdoc). We don't, which is OK relative to what we used to provide. We should probably improve in the future if jsdoc becomes the source of types for significant, typed codebases.
| } | ||
| else { | ||
| if (!jsdocTypeLiteral.jsDocPropertyTags) { | ||
| jsdocTypeLiteral.jsDocPropertyTags = [] as MutableNodeArray<JSDocPropertyTag>; |
There was a problem hiding this comment.
See earlier comment on casting to MutableNodeArray.
| if (!typedefTag.jsDocTypeLiteral) { | ||
| typedefTag.jsDocTypeLiteral = <JSDocTypeLiteral>typeExpression.type; | ||
| if (jsdocTypeLiteral) { | ||
| if (typeExpression && typeExpression.type.kind === SyntaxKind.ArrayType) { |
There was a problem hiding this comment.
What are our performance guidelines on optionally vs unconditionally assigning to a property?
There was a problem hiding this comment.
I thought the standard for booleans was to declare them as b?: boolean and only assign them when needed. What difference in performance could arise? Conditional assignment saves space (but how much I don't know) versus unconditional assignment, which could be better for [de]optimisation. @rbuckton do you have an opinion?
| } | ||
| } | ||
|
|
||
| function textsEqual(parent: EntityName, name: EntityName): boolean { |
There was a problem hiding this comment.
Nit: would just name these variables a and b since they don't really depend on a parent-child context.
| while (token() !== SyntaxKind.EndOfFileToken) { | ||
| nextJSDocToken(); | ||
| switch (token()) { | ||
| case SyntaxKind.AtToken: |
| const child = tryParseChildTag(target); | ||
| if (child && child.kind === SyntaxKind.JSDocParameterTag && | ||
| (ts.isIdentifier(child.fullName) || !textsEqual(fullName, child.fullName.left))) { | ||
| break; |
There was a problem hiding this comment.
Should this be break loop? It looks like the way it is currently, it goes all the way to the end of the file, when we could stop the moment we see something that isn't an @param tag with the correct name.
There was a problem hiding this comment.
This function is always called from in a speculative context, so I can actually just return false.
| break; | ||
| } | ||
| } | ||
| scanner.setTextPos(resumePos); |
There was a problem hiding this comment.
Do we have an established pattern for re-setting the scanner? What if scanner has some other state that this doesn't reset?
There was a problem hiding this comment.
Yep, this is left over from the old scanChildTags that faked its own speculative parsing. Removed.
| scanner.setTextPos(resumePos); | ||
| } | ||
|
|
||
| function tryParseChildTag(target: PropertyLikeParse, alreadyHasTypeTag?: boolean): JSDocTypeTag | JSDocPropertyTag | JSDocParameterTag | false { |
There was a problem hiding this comment.
This is only called in one place, and alreadyHasTypeTag is never provided.
There was a problem hiding this comment.
Oops, I moved that up a couple of levels and forgot to remove it here.
| // Error parsing property tag | ||
| return false; | ||
| return target === PropertyLikeParse.Property && parseParameterOrPropertyTag(atToken, tagName, target); | ||
| case "arg": |
There was a problem hiding this comment.
Is the information about what tag name was used available in the AST? That might be useful for linting.
There was a problem hiding this comment.
Yes, JSDocTag.tagName: Identifier, so it looks like it's preserved on all JSDocTags.
A more useful lint rule would be to look for mismatched typedef/param and param/property nestings, since the compiler currently ignores them:
/**
* @typedef {object} x
* @param {string} x.a - WRONG! (but ignored by the compiler)
*
* @param {object} y
* @property {string} y.a - WRONG! (but ignored by the compiler)
*/| return currentToken = scanner.scanJSDocToken(); | ||
| } | ||
|
|
||
| function parseJSDocEntityName(createIfMissing = false): EntityName { |
There was a problem hiding this comment.
This is only called in one place, and createIfMissing is always true.
There was a problem hiding this comment.
This could use:
function createQualifiedName(entity: EntityName, name: Identifier): QualifiedName {
const node = createNode(SyntaxKind.QualifiedName, entity.pos) as QualifiedName;
node.left = entity;
node.right = name;
return finishNode(node);
}And you can re-use createQualifiedName in parseEntityName. (parseJSDocEntityName should probably be near parseEntityName since they are similar.)
There was a problem hiding this comment.
Done. I thought about merging parseEntityName and parseJSDocEntityName, but they both worry about ad-hoc jsdoc-related things, plus they call into different identifier parsers which do the same.
| pushClassification(tag.postParameterName.pos, tag.postParameterName.end - tag.postParameterName.pos, ClassificationType.parameterName); | ||
| pos = tag.postParameterName.end; | ||
| if (!tag.isParameterNameFirst) { | ||
| pushCommentRange(pos, tag.name.pos - pos); |
There was a problem hiding this comment.
These two if blocks are the same, so use this helper:
function pushName(name: Identifier) {
pushCommentRange(pos, name.pos - pos);
pushClassification(name.pos, name.end - name.pos, ClassificationType.parameterName);
pos = name.end;
}There was a problem hiding this comment.
Don't know why this shows as outdated, it still seems relevant.
|
|
||
| export function getJSDocParameterNameCompletions(tag: JSDocParameterTag): CompletionEntry[] { | ||
| const nameThusFar = unescapeLeadingUnderscores(tag.name.text); | ||
| const nameThusFar = isIdentifier(tag.fullName) ? unescapeLeadingUnderscores(tag.name.text) : undefined; |
There was a problem hiding this comment.
I think we should bail out immediately if it's not an identifier, instead of continuing with an undefined nameThusFar.
There was a problem hiding this comment.
Done. I returned emptyArray instead of undefined. I'm not sure whether that's correct.
I'll take a look at Wesley's next and see if those require any changes.
sandersn
left a comment
There was a problem hiding this comment.
Responses to review comments
| return declareSymbolAndAddToSymbolTable(node as JSDocPropertyTag, | ||
| (node as JSDocPropertyTag).isBracketed || ((node as JSDocPropertyTag).typeExpression && (node as JSDocPropertyTag).typeExpression.type.kind === SyntaxKind.JSDocOptionalType) ? | ||
| return declareSymbolAndAddToSymbolTable(node as JSDocPropertyLikeTag, | ||
| (node as JSDocPropertyLikeTag).isBracketed || ((node as JSDocPropertyLikeTag).typeExpression && (node as JSDocPropertyLikeTag).typeExpression.type.kind === SyntaxKind.JSDocOptionalType) ? |
| visitNode(cbNode, (<JSDocTypedefTag>node).fullName) || | ||
| visitNode(cbNode, (<JSDocTypedefTag>node).name) || | ||
| visitNode(cbNode, (<JSDocTypedefTag>node).jsDocTypeLiteral); | ||
| if ((node as JSDocTypedefTag).typeExpression && |
| postParameterName?: Identifier; | ||
| typeExpression: JSDocTypeExpression; | ||
| /** Whether the property name came before the type -- non-standard for JSDoc, but Typescript-like */ | ||
| isParameterNameFirst: boolean; |
| @@ -1553,6 +1553,10 @@ namespace ts { | |||
|
|
|||
| /** Does the opposite of `getJSDocParameterTags`: given a JSDoc parameter, finds the parameter corresponding to it. */ | |||
| export function getParameterFromJSDoc(node: JSDocParameterTag): ParameterDeclaration | undefined { | |||
| declaration = getDeclarationOfKind<TypeAliasDeclaration>(symbol, SyntaxKind.TypeAliasDeclaration); | ||
| type = getTypeFromTypeNode(declaration.type); | ||
| } | ||
| const declaration = getDeclarationOfKind<JSDocTypedefTag>(symbol, SyntaxKind.JSDocTypedefTag) || |
There was a problem hiding this comment.
I'll just use findDeclaration which takes a predicate.
|
| let seenAsterisk = false; | ||
| while (true) { | ||
| nextJSDocToken(); | ||
| switch (token()) { |
There was a problem hiding this comment.
Can we return false; in the default case? It looks like if we keep encountering some other kind of token, we'll go all the way to the end of the file.
There was a problem hiding this comment.
Not really; we hit the default case a lot for punctuation and identifiers that we want to skip as part of the tag's description.
Note that it's the end of the comment, not the end of the whole file. And we return false as soon as tryParseChildTag encounters any tag that is not param, type or property. So runaway parsing shouldn't be a big problem.
Also note: Comments aren't handled by this PR but once they are, default will probably look like parseJSDocCommentWorker, where it just pushes the token into a comment array.
|
I think this is ready to go unless you have more comments. |
|
Addressed the comment I missed previously. |
Now Typescript supports the creation of anonymous types using successive
@paramlines in JSDoc:This is equivalent to the Typescript syntax
{ s: string, n: number }, but it allows per-property documentation, even for types that only need to be used in one place. (@typedefcan be used for reusable types.)If the type of the initial
@paramis{object[]}, then the resulting type is an array of the specified anonymous type:Finally, nested anonymous types can be created by nesting the pattern:
Implementation notes:
os[].nested.length, but it doesn't check that usages are correct. Such checking would be easy to add but tedious and low-value.@typedefdoesn't support nested@propertytags, but does supportobject[]types. This is mostly a practical decision, backed up by the fact that usejsdoc.org doesn't document nested types for@typedef.Fixes #11597