Repository navigation
all: add opt-in transparent telemetry to Go toolchain #58894
Description
In February I posted a series of blog posts defining Transparent Telemetry, and we had a lively discussion on #58409. In the original posts, the design was opt-out. Based on that discussion as well as private discussions with long-time contributors and users, I revised the design to be opt-in.
I propose that we add opt-in transparent telemetry to the Go toolchain as described in those posts, specifically “The Design of Transparent Telemetry.” Transparent Telemetry has the following key properties:
- The decisions about what metrics to collect are made in an open, public process.
- The collection configuration is automatically generated from the actively tracked metrics: no data is collected that isn’t needed for the metrics.
- The collection configuration is served using a tamper-evident transparent log, making it very difficult to serve different collection configurations to different systems.
- The collection configuration is a cacheable, proxied Go module, so any privacy-enhancing local Go proxy already in use for ordinary modules will automatically be used for collection configuration.
- Uploaded reports only include total event counts over a full week, not any kind of time-ordered event trace.
- Uploaded reports do not include user IDs, machine IDs, or any other kind of ID.
- Uploaded reports only contain strings that are already known to the collection server: counter names, program names, and version strings repeated from the collection configuration, along with the names of functions in specific, unmodified Go toolchain programs for stack traces. The only types of non-string data in the reports are event counts, dates, and line numbers.
- IP addresses exposed by the HTTP session that uploads the report are not recorded with the reports.
- Thanks to sampling, only a constant number of uploaded reports are needed to achieve a specific accuracy target, no matter how many installations exist. Specifically, only about 16,000 reports are needed for 1% accuracy at a 99% confidence level. This means that as new systems are added to the system, each system reports less often. Exactly how often will depend on how many systems opt in.
- The aggregate computed metrics are made public in graphical and tabular form.
- The full raw data as collected is made public, so that project maintainers have no proprietary advantage or insights in their role as the direct data collector.
- The system is off by default and requires an explicit opt-in.
Please note, as described in the Why Telemetry? section of the intro post, that telemetry addresses a different kind of problem than bug reports and surveys. In particular, relying on bug reports is not sufficient to identify problems that don't obviously impact functionality, including performance problems, and surveys are not sufficient to identify the variety of usage and contexts where Go is used and which would inform prioritization of effort.
There is good reason to believe that with even tens of thousands of users opted in, we should be able to get helpful data. It will not be as complete as the opt-out system, but it should be good enough. As described in the Can We Still Make Good Decisions? section of the opt-in post, there will be certain biases in the data based on who is more likely to opt in. Once we have data, it would make sense to compare “technical demographics” like operating system and editor against the annual Go survey and Stack Overflow surveys. If there is significant skew, we could look into reweighting the data as standard polls do (https://en.wikipedia.org/wiki/Iterative_proportional_fitting).
For examples of the kinds of questions we'd use telemetry to answer and the kinds of decisions those answers would inform (but not decide directly), see “Use Cases for Transparent Telemetry”.
Activity
rsc commentedon Mar 8, 2023
This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group
willfaught commentedon Mar 9, 2023
@rsc I didn't get a chance to ask this before the discussion was closed:
Why not assign every known string a number, and then use those numbers instead of sending strings? While sending the strings themselves is equivalent to the numbers, there's a "slippery slope" whiff I get from it. Publishing the known string index is another way to assure users that their privacy is being protected. There's an extra feeling of security knowing that only numbers are being sent.
Merovius commentedon Mar 9, 2023
@willfaught I actually think it would be less transparent, not more. Because instead of seeing the actual human-readable information that is sent, you get a bunch of cryptic numbers that you have to interpret. Even if it was increasing a "feeling of privacy", there is a good argument to be made that giving such a feeling without actually increasing privacy (which this wouldn't do) is a bad thing. This would be privacy theater, if you will. Combine that with the fact that this would be significantly more churn and work - you either have the mapping be automatic, in which case it might completely re-number everything on every Go release, or you have to manually keep it up to date whenever something changes - and this seems an overall pretty bad idea.
alvarolm commentedon Mar 9, 2023
Thank you for:
allowing anyone to participate in this this conversation
listening to the community
making improvements based on public feedback
So far I see a thoughtfully and polished proposal with user privacy as a fundamental guideline.
Currently I'm prone to contribute and participate opting in, as this is a clear and reasonable way to help furthering the language development.
I trust the spirit of the proposal that safeguards the balance between privacy and functionality is preserved and never broken.
I'm making these comments as simple user who has chosen to use the language for work and personal use for many years now, not having any significant reason to doubt or question this choice.
Grateful for the work being done and the direction where the language is going.
willfaught commentedon Mar 9, 2023
Most of the data sent will be counters/numbers anyway. The format is binary, if I remember correctly, so it can be mmap'ed. This wouldn't affect the human readability of the data either way.
Avoiding a slippery slope isn't theater.
In my opinion, there isn't much widespread sympathy for having to do more work to do the right, best thing in terms of privacy. This change in direction happened because the community insisted that the Go team do the harder thing. Russ acknowledged that this change will require more work:
So I don't think it's necessarily relevant how hard it is, not without a careful tradeoff analysis. And I doubt it would entail as much hassle as you make it out to be, once a process is in place. For instance, the index can be immutable, and added to over time and releases with an AST scan. There may indeed be a good reason not to do this, but I'm not convinced that this is one of them.
thepudds commentedon Mar 9, 2023
Hi @willfaught, there is a binary file used for performance reasons, but as I understand it, the uploaded data is purposefully in human-readable JSON. From the design post:
If the Go toolchain was to be underhanded about what it is doing, information could in theory be smuggled in a series of words or a series of numbers.
To reduce the chance of the Go toolchain doing something underhanded here, including regarding how it handles strings, I suspect someone opting in will need to rely at least in part on the Go toolchain being open source (and either look at the code themselves, or rely on the likelihood of others doing so, the reproducibility of toolchain builds, the code review process, etc.).
If someone crosses that threshold of trust and is willing to choose to opt in, I tend to agree with @Merovius that plain text strings are easier to understand and help with transparency more than a series of numbers.
AndrewHarrisSPU commentedon Mar 9, 2023
From https://research.swtch.com/telemetry-design:
I think this is saying the counter files are locally compact, but emitted as JSON. In the original thread someone made the point that plaintext is a good clue to someone who is sniffing traffic what's going on, that made sense to me. In a wonky way, a set of plaintext strings is already a set of unique numbers in this kind of scenario.
willfaught commentedon Mar 10, 2023
I see. It's hard to judge the readability of
{"foo": 12}vs.{"46": 12}without knowing what foo might be, but if the string index is published, people can work out for themselves that{"46": 12}means{"error_when_build_stdlib": 12}. Most of the time, nobody is going to be looking at these, especially if they have numbered keys, so optimizing their readability doesn't seem that important. It would be trivial to write a little tool people could use to translate the string IDs in what was sent into the strings themselves, if they want to know.I think the bigger concern—bigger than the malicious sending of personal data strings—is the accidental sending of personal data strings. Nobody wants "home_video_of_[embarrassing thing].mov" or "[embarrassing thing]website.com/go-foo/bar" accidentally sent to the Go team. And perhaps no one will file an issue to fix it either, out of embarrassment, when the problem is discovered. Not sending any strings at all rules out that possibility entirely.
ianlancetaylor commentedon Mar 10, 2023
By design only strings that are explicitly requested by the telemetry server will be sent to the telemetry server. Of course we can't stop anyone from modifying the Go tools to send other strings, but it hardly seems likely to happen by accident.
doggedOwl commentedon Mar 10, 2023
@willfaught publishing numbers instead of strings is a false sense of security. It accomplishes nothing. Who does it protect from when the strings are public and can be consulted anytime by anyone? Users would be more comfortable reading and understating right there what is being sent instead of cross-referencing that is a burden for the user an no burden for any would-be malicious actors.
As it is, with the change to the Opt-in Mechanism, this is a very good balanced approach.
37 remaining items
gopherbot commentedon May 1, 2024
Change https://go.dev/cl/582695 mentions this issue:
cmd/internal/telemetry: add NewStackCounter functionscmd/internal/telemetry: add NewStackCounter functions
gopherbot commentedon May 8, 2024
Change https://go.dev/cl/584276 mentions this issue:
cmd/link: add telemetry supportcmd/compile: add support for telemetry
cmd/link: add support for telemetry
gopherbot commentedon May 13, 2024
Change https://go.dev/cl/585235 mentions this issue:
cmd: add telemetry for commands in cmdcmd: add telemetry for commands in cmd
gopherbot commentedon May 16, 2024
Change https://go.dev/cl/586138 mentions this issue:
cmd/go: add telemetry for a predefined set of GOROOT valuescmd/go: add telemetry for a predefined set of GOROOT values
findleyr commentedon Aug 14, 2024
@matloob with the 1.23 release, is this done?
matloob commentedon Aug 14, 2024
Yes.