TAP 1.6 - Artifact Metadata Repository Observer

As part of the new version of the scanning mechanism in TAP which was released in Alpha in version 1.5, and has now been promoted to Beta in TAP 1.6, we now have a new component called the Artifact Metadata Repository Observer (AMR Observer). Overview This new component is part of the Artifact Metadata Repository (AMR), a service designed to be a central source of truth for artifact metadata across multiple clusters and environments. AMR Observer watches for changes in resources and emits cloud events when these changes occur. It enhances the capabilities of the AMR by providing real-time updates about the state of resources. The AMR Observer brings several key features to TAP 1.6: Real-time Monitoring: The AMR Observer watches for changes in specific resources and emits cloud events when these changes occur. This allows for real-time monitoring and tracking of resources. Customizable Configuration: Users can configure the AMR Observer to watch for changes in specific resources. This customization allows users to focus on the resources that are most relevant to their needs. Integration with AMR: As a component of the AMR, the AMR Observer contributes to the AMR’s goal of providing a single, consistent view of artifact metadata across multiple clusters and environments. The AMR Observer is deployed to the build and run clusters when enabled. It communicates with the Kubernetes API Server to obtain the cluster’s location ID and emits a cloud event to the AMR Cloud Event Handler. The AMR Observer watches for ImageVulnerabilityScans and workload ReplicaSets. Why Does This Matter As part of the redesign of the scanning mechanisms within TAP, one of the key goals was to simplify the extensibility of the platform to allow for a Bring Your Own Scanner (BYOS) model in a much more simplified and straight forward manner. While TAP has always allowed for you to bring your own scanner, the UX for this was not great to say the least, and required deep knowledge and tight coupling with TAP itself, making the process not very approachable for most customers. When scanning an image or source code, 4 key steps must happen: Perform the scan Output an SBOM in CycloneDX or SPDX format Push the data to the central metadata store validate the scan results against your desired security policy In the old version of the Scanning architecture, when bringing your own scanner, you needed to solve all of these issues in one resource, causing confusion and a great dealk of overhead. In the new AMR model, you only need to deal with steps 1 and 2, while the platform will automatically handle the rest for you. AMR Observer is the new component which is in charge of the 3rd step, which basically means that it will watch for scan CRs in your cluster, and when the occur, it will retrieve the outputted SBOM which is now stored beyond in the metadata store, also as an OCI artifact in your container registry of choice, and normalize the data and send it to the AMR metadata store via cloud events. This is only a small part of the AMR observer, but it is a huge improvement of the current situation, decoupling the scanning from the platform specific needs, makes integrating new scanners much easier and provides many other security benefits as well. The other thing mentioned above which AMR Observer watches are the replicasets of your clusters, and in particular, those of your deployed workloads. When a change happens in a workload, for example a new revision is being deployed, the AMR Observer will report this data to AMR as well via cloud events. this allows us to have a single place of truth, where we can understand what is running where at any given time! While currently the data collected by AMR is relatively limited, the foundation is there for more and more relevant data to be added in future releases which I am very excited about. By having a central location which can aggregate data from CI, CD and running applications, as well as the ability to correlate these pieces of data, we can achieve truly amazing insights, we can perform simple correlation between a source and the final running app in production, and understand historically what it went through in order to get there, which means that we can also use this data to understand how a platform like TAP is assisting us in improving our developer experience, and velocity using a mechanism like DORA metrics or other similar standards. Another key element of AMR, is the GraphQL endpoint which is exposed in TAP 1.6, allowing us to not only get the data to a central location, but also query that data based on our own unique needs, using a simple and user friendly GraphQL playground which is included in TAP 1.6! Because this endpoint is now exposed to us, we can also use it to build out dashboards in external monitoring solutions such as Grafana. We can see for example bellow an example of a widget i have built in Grafana, to show based on the location which is the runtime cluster, which revisions of apps are running, and additional information about them: While not all data is exposed here, with the foundation in place, im truly excited to see where this goes in the future releases! Summary While the new scanning mechanism is not fully at feature parity with the original mechanism, the new features and design including the AMR Observer, are extremely promising in my mind, and once they mature a bit more they will truly provide a much better offering to TAP users than what we currently have! This new architecture and what it can bring to the table is truly exciting from my perspective, as it allows for not only a better experience within the platform itself, with more advanced functionality, but also allows for easy integration into thrid party systems, using the GraphQL endpoint!