Hosting Internal Proxy and Internal pkg.go.dev for Internal Library’s Documentation
How we host our own internal Proxy and internal pkg.go.dev at Xendit
Hey everyone, it’s been a while I haven’t published any blog posts. I think it’s been 6 months already haha. I was so busy for the past few months! There were just too many things happening and it kept me out busy from writing.
So here today, I’m sharing with you all about one of my experiments last year. Yes, it was last year. But hopefully, the content that I want to share here is still relevant and will help you with the pain I had previously in my job. The topic will be about hosting our own pkg.go.dev for our internal libraries at Xendit
Background
It’s not a secret, most of the companies have their own private libraries that will only be used for internal usage. It could be for internal logging, internal tracing, internal error libraries, or anything that could satisfy their systems needs.
It is the same for us at Xendit. My current team, DevPlatform, is responsible to help the engineer to be able to move faster. Thus, we want to help our product engineer work faster by providing and maintaining a lot of tools and libraries that are commonly used in every codebase. For instance, we built CLI Generator for some code templates, Logger libraries, and other common tools, both are written in NodeJS and Golang.
Specifically for Golang, we have various internal libraries for internal usages. In the beginning, we only had 2–3 Go libraries that were not hard to maintain. But as the team grows, the needs for Go libraries also grow bigger. We started to build more and more libraries until we realized, we already have a handful number of libraries written in Go.
Looking at the situation, we then observed a few problems that can cause the developer toil as the team growing. We want to solve these problems earlier so that our engineers will have a positive experience working with our Go internal libraries.
Problem 1: Looking for Documentation Website for Private Libraries
As a Gopher, I’ve been using pkg.go.dev since its first release. It’s a portal website for public Golang libraries. It really helps to see the detail of libraries without having to check the source code. It will generate the documentation site for any public libraries that are available in Github or any public Git provider (Gitlab, Bitbucket). So let’s say, I want to see what are the available functions in the Logrus library, I can easily see the complete functions in the pkg.go.dev like below.
Having realized that, I see that pkg.go.dev is only for public libraries. Yet for private/internal libraries, it is not available there. Because of that, I immediately try to figure out how we can host our internal documentation sites.
After looking for the solutions, I finally found pkg.go.dev website is already open-sourced here https://github.com/golang/pkgsite. That’s where we dig deeper in our exploration.
Problem 2: Easily Pull Private Libraries without Complex Configuration
Another problem we start to realize is, the engineering onboarding process to use Golang is quite longer compared to onboard them to use Node JS. For the Node JS project, we only need to create the .npmrc (a configuration for private NPM) then you will be able to pull all internal libraries.
But for Golang on the other hand, there isn’t anything like .npmrc for configuring the private libraries registry. Hence, it’s difficult, especially for new Gopher engineers. Sometimes, the error can be like this
$ go get github.com/xendit/internal-library-namehttps://sum.golang.org/lookup/github.com/xendit/internal-library-name@v0.0.0-20190921175342-61a76c096369: 410 Gone
Or sometimes something like this
$ go get github.com/xendit/internal-library-name
# cd .; git ls-remote https://github.com/xendit/internal-library
fatal: could not read Username for 'https://github.com': terminal prompts disabledEven if you’re an experienced Golang engineer, solving this could be tricky. You either need to do:
- Configure your local
id_rsato download from Github through ssh. - Setting your
GOPRIVATEenvironment. So it will be pointing to the Github repo directly. - Or by using your personal Github Token to pull the private libraries.
And as a Devplatform engineer, we realized this is could be a problem. We then see that having a private proxy could help the engineers to download/pull their private libraries without the need to configure a lot of things in their local.
They will only need to:
- Connect to our internal VPN
- And set the GOPROXY environment
And they will be able to download all internal libraries.
And because of these 2 problems (easy to pull library and internal documentation site), we then decide to host our own internal proxy and documentation site in our own infrastructure.
pkg.go.dev Architecture
On the Github repo of the pkgsite, we can see how the pkgsite work. This is the complete architecture of how pkgsite works behind the scene.
That’s quite complicated works. When I see this for the first time, I was overwhelmed, like, WTF! The time that I’ll spend on maintaining all of this is not worth the impact on the company.
Then I decided to spend a few weeks doing research on this in parallel. Whenever I was free, I tried to digest the code and the architecture, reading the docs, even asking many Go experts.
After having a lot of discussions with many people, I came to a basic conclusion. To host an internal pkgsite, we don’t have to deploy all those stuff on that diagram.
In fact, we only need the following two items
- The frontend application > we can get it from the
pkgsiterepo - The Proxy server
The Frontend Application
The frontend app can be seen on the pkgsite repository. To deploy this is simple; since we’re using the Kubernetes, we only need to create the Dockerfile and the deployment and service config in the Kubernetes. Moreover, we need to register the domain to the internal domain.
This is how our Dockerfille looks like,
From the Dockerfile, as we can see, we pull the code of the pkgsite directly and compile it for docker distribution. Originally this Dockerfile is copied from Micah Parks’s Github repo (https://github.com/MicahParks/private-pkgsite). You might want to explore deeper, feel free to raise a PR or issue on that repository.
The Proxy Server
Since the pkg.go.dev is too huge to be hosted for internal needs, the only option we can think of is by using the proxy server.
We found a few candidates for the proxy server. But after trying it, we decided to use Athens as our proxy.
The reasons for this were:
- It’s configurable, we can use custom DB, custom cache
- It is used by a lot of companies. We believe it will be easier to debug and asking about issues in the communities.
- The maintainers are active and a part of the Gopher Slack community. So if we found an issue, we can ask them directly.
To draw the architecture on how we use Athens from end-to-end, you can see the diagram below.
To deploy the proxy server turned out to be pretty fast, we only need to follow the documentation.
This is our Dockerfile example.
For the deployment, we deployed it on top of Kubernetes. Since our proxy is already dockerized, it helps us to deploy it to our infrastructure.
More documentation on how to deploy Athens proxy in the internal systems can be seen here. It may differ depending on the cloud/environment you use.
Security Concerns
Since the repo doesn’t have anything related to how to handle authentication. To avoid the internal libraries leaked to the public, the only solution we choose is by protecting the domain under an internal VPN.
Both documentations and Proxy servers are only accessible through VPN.
How to Use
For internal gopher engineers, if they want to use the Proxy, it will be really straightforward. Like I said previously on top of this article, they will only need to:
- Connect to our internal VPN
- And set the GOPROXY, and GONOSUMDB environment
$ go env -w GOPROXY="https://proxy.golang.org,https://go-proxy.xendit.com,direct"$ go env -w GONOSUMDB="github.com/xendit/*"
Some important notes:
- The order for the proxy value in
GOPROXYmatters. From the example above, it means, the Go application will pull the library first from the public Golang proxy, then go to our internal proxy, and if the library still does not exist, it will go directly to Github. - When we use
GOPROXY,we need to removeGOPRIVATEvalue. By default, the go application will look from theGOPRIVATEfirst. So the proxy won’t be used if theGOPRIVATEvalue is set.
After that, we will be able to download all internal libraries through the internal proxy.
Conclusions
After deploying both for Proxy and Documentation site(pkgsite) in our internal systems. We are now able to download internal libraries without complex setup in our local. Moreover, all of our private libraries are now available in our internal documentation site (pkgsite). The engineers are now able to see the documentation of all libraries with the same experience as any other public library.
The time that was needed to explore all these things took me around 1 month. Thanks to the infra engineer as well, he helped me to experiment on this. But I think if we do it intensively, it should take only a few days to work on this. I did not manage to finish it faster because I only worked on this during my spare time. So there’s no urgency and focus time ;). Although the priority was low, this documentation & library system is still helpful for future development and our internal Go ecosystem.
What’s next?
We’re planning to explore the same thing with TS/JS for our Node ecosystem at Xendit. The key is to reduce any developer toils as much as possible.
References and Thanks
- Micah Parks (https://github.com/MicahParks/private-pkgsite)
- Patrick Juen (our Infra engineer that helps me to explore this)
- Stanley Nguyen for PRs review and design review
