Re: [community-group] Define order of operations during parsing of tokens files (#123)

You've raised a really interesting point that we've not really addressed at all.

In one of our earliest drafts (before it was even public) we were considering some kind of import mechanism, where one token file could import another. But we quickly realised there's quite a lot of complexity to resolve with an approach like that. For example, if token file A imports token file B, and a tool reads token file A...

- does it see a combined set of all tokens from files A and B?
   - or does it only see tokens from file A (but perhaps tokens in A can reference tokens in file B)?
- if there are tokens with the same name in files A and B is that:
    - an error?
    - or does the token from file A override the one from B
- what a about groups with the same name in files A and B - do they get merged? Or does one replace the other?
- if tokens in one file can override tokens from another file, do their types need to be the same? If not, what are the consequences for other tokens that reference that token?

In the interest of keeping our version 1 spec simple, we decided to drop the idea for the time being. I think there was a hope/assumption that tools would solve this somehow.

But, as shown in your example, that does raise an interesting question when it comes to references. Is a token that references another which does not exist in the same file valid? If you take the view that, since the spec says nothing about working with multiple token files, each token file must be self-contained, then I'd say that should not be valid. But overriding some tokens is desireable for use-cases like theming. And being able to split very large sets of tokens over several files is also desirable. So there probably should be an official way for a token in one file to reference a token in another.

My personal preference would be to revisit the import idea. That would put the onus on the spec to clearly define what the behaviour should be which will benefit interoperability between tools. I think it would also help make the order in which files are being included explicit.

To encourage more discussion, here's a _rough_ proposal of how this _could_ work...

`file1.tokens.json` (a self-contained tokens file, where all references must point to tokens in the same file):

```json
{
  "token-a": {
    "$value": "#123456",
    "$type": "color"
  },
  
  "group-b": {
     "token-b-1": {
      "$value": "1.5rem",
      "$type": "dimension"
    }
  },
  
  "alias-token-c": {
    "$value": "{group-b.token-b-1}"
  }
}
```

`file2.tokens.json` (another self-contained tokens file, where all references must point to tokens in the same file):

```json
{
  "token-a": {
    "$value": "#abcdef",
    "$type": "color"
  },
  
  "group-b": {
     "token-b-2": {
      "$value": "320ms",
      "$type": "duration"
    }
  },
  
  "alias-token-d": {
    "$value": "{group-b.token-b-2}"
  }
}
```

`file3.tokens.json` (which includes `file1` & `file2`. Tokens in `file3` are therefore allowed to reference tokens in `file1` and `file2`):

```json
{
  "$includes": [
    "./path/to/file2.tokens.json",
    "https://design-system.example.com/tokens/file3.tokens.json"
  ],
  
  "alias-token-c": {
    "$value": "{token-a}"
  }
}
```

The behaviour I would suggest when parsing `file3` is:

* Files listed in the `$includes` array are loaded, parsed and then deep merged into the current file
* Where tokens have the same name, the order of precedence is: The file that is including the others followed by the files listed in the `$includes` array in reverse order.

So, in this example: tokens in `file3`  override tokens in `file2`, which in turn override tokens in `file1`.

Therefore, the end result is equivalent to a single file like this:

```jsonc
{
  // token-a in file2 overrides token-a in file1, so
  // the value is #abcdef
  "token-a": {
    "$value": "#abcdef",
    "$type": "color"
  },

  // Since group-b exists in both file1 and file2, a
  // merged version of those is added here:
  "group-b": {
     // this token comes from file1
     "token-b-1": {
      "$value": "1.5rem",
      "$type": "dimension"
    },

    // this token comes from file2
    "token-b-2": {
      "$value": "320ms",
      "$type": "duration"
    }
  },
  
  // alias-token-c in file3 overrides alias-token-c in file1
  // so it references token-a.
  // Therefore, its resolved value is #abcdef
  "alias-token-c": {
    "$value": "{token-a}"
  }
  
  // this token comes from file2
  "alias-token-d": {
    "$value": "{group-b.token-b-2}"
  }
}
```

Thoughts?

-- 
GitHub Notification of comment by c1rrus
Please view or discuss this issue at https://github.com/design-tokens/community-group/issues/123#issuecomment-1090883542 using your GitHub account


-- 
Sent via github-notify-ml as configured in https://github.com/w3c/github-notify-ml-config

Received on Wednesday, 6 April 2022 22:39:51 UTC