什么是可观测 相信大家日常经常使用kibana
、grafana
、jaeger
等平台观察系统状态和定位问题,这些就是可观测体系的一部分。可观测主要包括:
日志,实现有ELK(es/logstash/kibana)
等
指标,实现有grafana
+promthues
等
追踪,基于opentracing
协议的实现有jaeger
、skywalking
等
我们排查问题过程,一般都会把三者日志、指标、追踪结合来看,比如通过接口异常增量指标发现问题—>链路追踪定位异常服务—>排查异常服务日志,所以关于可观测我们经常可以看见这个经典的图片:
什么是OpenTelemetry
关于可观测除了上述的各种实现外,还有另一套实现OpenCensus
,于是诞生统一标准OpenTelemetry
且兼容OpenTracing
,OpenCensus
。不过关于go语言OpenTelemetry
的统一sdk实现还不完善,比如目前还不支持日志,具体可以查看https://github.com/open-telemetry/opentelemetry-go
。
接下来,我们基于Go来看看,原sdk(也就是未使用OpenTelemetry
)接入指标和追踪的方式和基于OpenTelemetry
新体系的指标和追踪接入方式区别。
可观测之指标 基于原生promethues
sdk的指标采集演示
Go版本 这里用的1.14
主要依赖的包:
"github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp"
就依赖了两个包,使用比较简单:
单独创建一个server使用github.com/prometheus/client_golang/prometheus/promhttp
暴露指标
使用github.com/prometheus/client_golang/prometheus
创建自定义指标,比如NewCounterVec
创建计数器、HistogramVec
创建直方图等等。
WithLabelValues
给自定义指标打标签
代码示例如下:
docker示例:https://github.com/TIGERB/easy-tips/tree/master/docker/grafana-promethues/go-demo/main.go.promethues
package mainimport ( "net/http" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" ) var ( GlobalApiCounter *prometheus.CounterVec ) func init () { GlobalApiCounter = prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "demo_api_request_counter" , Help: "接口请求次数自定义指标" , }, []string {"domain" , "uri" }, ) prometheus.MustRegister(GlobalApiCounter) } func main () { go (func () { metricServer := http.NewServeMux() metricServer.Handle("/metrics" , promhttp.Handler()) http.ListenAndServe(":2112" , metricServer) })() http.HandleFunc("/v1/demo" , func (w http.ResponseWriter, r *http.Request) { GlobalApiCounter.WithLabelValues(r.Host, r.RequestURI).Inc() w.Write([]byte ("test" )) }) http.ListenAndServe(":6060" , nil ) }
访问接口curl 127.0.0.1:6060/v1/demo
后,查看指标输出 curl 127.0.0.1:2112/metrics
,如下:
# HELP demo_api_request_counter 接口请求次数自定义指标 # TYPE demo_api_request_counter counter demo_api_request_counter{doamin="127.0.0.1:6060",uri="/v1/demo"} 4
基于OpenTelemetry
sdk的promethues
指标采集演示
Go版本 这里用的1.19 版本太低报错
主要依赖的包:
"github.com/prometheus/client_golang/prometheus/promhttp" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/prometheus" "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/sdk/metric"
相对于原生prome只使用两个包,引入的包多几个:
相同点:仍然 单独创建一个server使用github.com/prometheus/client_golang/prometheus/promhttp
暴露指标
不同点:使用go.opentelemetry.io/otel/exporters/prometheus
初始化一个指标对象meter
不同点:使用meter.Int64Counter
初始化计数器、直方图等
不同点:metric.WithAttributes
打标签
代码示例如下:
docker示例:https://github.com/TIGERB/easy-tips/tree/master/docker/grafana-promethues/go-demo/main.go
package mainimport ( "context" "net/http" "github.com/prometheus/client_golang/prometheus/promhttp" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/prometheus" "go.opentelemetry.io/otel/metric" metricsdk "go.opentelemetry.io/otel/sdk/metric" ) var meter metric.Meterfunc init () { mexp, err := prometheus.New() if err != nil { panic (err) } meter = metricsdk.NewMeterProvider(metricsdk.WithReader(mexp)).Meter("http-demo" ) } func main () { urlCouter, _ := meter.Int64Counter("demo_api_request_counter" , metric.WithDescription("QPS" )) go (func () { metricServer := http.NewServeMux() metricServer.Handle("/metrics" , promhttp.Handler()) http.ListenAndServe(":2112" , metricServer) })() http.HandleFunc("/v1/demo" , func (w http.ResponseWriter, r *http.Request) { opt := metric.WithAttributes(attribute.Key("domain" ).String(r.Host), attribute.Key("uri" ).String(r.RequestURI)) urlCouter.Add(context.Background(), 1 , opt) w.Write([]byte ("test" )) }) http.ListenAndServe(":6060" , nil ) }
访问接口curl 127.0.0.1:6060/v1/demo
后,查看指标输出 curl 127.0.0.1:2112/metrics
,如下:
# HELP demo_api_request_counter_total QPS # TYPE demo_api_request_counter_total counter demo_api_request_counter_total{domain="127.0.0.1:6060",otel_scope_name="http-demo",otel_scope_version="",uri="/v1/demo"} 1
可观测之追踪 基于OpenTracing
sdk的jaeger
追踪演示
Go版本 这里用的1.18
主要依赖的包:
"github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing" "github.com/opentracing/opentracing-go" "github.com/uber/jaeger-client-go" "github.com/uber/jaeger-client-go/transport"
使用方式:
使用jaeger.NewTracer
创建一个tracer对象tracer
,并指定jaeger服务地址(这里采用的是非agent方式演示)
使用tracer
创建一个spantracer.StartSpan
,并在你想追踪的代码段后面span.Finish()
这里上游使用grpc
服务做测试,使用opentracing中间件SDK go-grpc-middleware/tracing/opentracing
docker示例:https://github.com/TIGERB/easy-tips/tree/master/docker/go-jaeger
package mainimport ( "context" "http-demo/demov1" "io" "net/http" "google.golang.org/grpc" grpc_opentracing "github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing" opentracing "github.com/opentracing/opentracing-go" jaeger "github.com/uber/jaeger-client-go" "github.com/uber/jaeger-client-go/transport" ) var ( tracer opentracing.Tracer ) func main () { sender := transport.NewHTTPTransport( "http://go-jaeger-jaeger-demo:14268/api/traces" , ) var closer io.Closer tracer, closer = jaeger.NewTracer( "http-demo" , jaeger.NewConstSampler(true ), jaeger.NewRemoteReporter(sender), ) defer closer.Close() http.HandleFunc("/v1/demo" , func (w http.ResponseWriter, r *http.Request) { span := tracer.StartSpan("demo_span_1" ) defer span.Finish() name, err := demoGrpcReq() if err != nil { w.Write([]byte (err.Error())) } w.Write([]byte (name)) }) http.ListenAndServe(":6060" , nil ) } func demoGrpcReq () (string , error ) { conn, err := grpc.Dial("grpc-demo:1010" , grpc.WithInsecure(), grpc.WithUnaryInterceptor(grpc_opentracing.UnaryClientInterceptor( grpc_opentracing.WithTracer(tracer), ))) if err != nil { return "" , err } defer conn.Close() client := demov1.NewGreeterClient(conn) resp, err := client.SayHello(context.TODO(), &demov1.HelloRequest{ Name: "http request" , }) if err != nil { return "" , err } return resp.GetMessage(), nil }
使用docker-compose up -d
启动演示服务,模拟请求curl 127.0.0.1:6060/v1/demo
,查看jaeger后台http://localhost:16686/
,示例如下:
基于OpenTelemetry
sdk的jaeger
追踪演示
Go版本 这里用的1.19 版本太低报错
主要依赖的包:
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/jaeger" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" "go.opentelemetry.io/otel/sdk/trace" "go.opentelemetry.io/otel/semconv/v1.17.0"
使用方式:
使用"go.opentelemetry.io/otel/exporters/jaeger"
初始化追踪tracer,同样指定jaeger服务地址(这里采用的是非agent方式演示)
http服务进行链路追踪,使用包go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
.NewHandler
创建http handler
grpc服务进行链路追踪,使用grpc otel追踪sdk go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
docker示例:https://github.com/TIGERB/easy-tips/tree/master/docker/go-otel
package mainimport ( "context" "http-demo/demov1" "net/http" "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" "google.golang.org/grpc" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/jaeger" "go.opentelemetry.io/otel/propagation" "go.opentelemetry.io/otel/sdk/resource" tracesdk "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.17.0" ) var tracer *tracesdk.TracerProviderfunc init () { exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint("http://jaeger-demo:14268/api/traces" ))) if err != nil { panic (err) return } tracer = tracesdk.NewTracerProvider( tracesdk.WithBatcher(exp), tracesdk.WithResource(resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceNameKey.String("http-demo" ), )), ) } func main () { otel.SetTracerProvider(tracer) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) ctx, cancel := context.WithCancel(context.Background()) defer cancel() defer tracer.Shutdown(ctx) demoHandler := func (w http.ResponseWriter, r *http.Request) { name, err := demoGrpcReq() if err != nil { w.Write([]byte (err.Error())) } w.Write([]byte (name)) } otelHandler := otelhttp.NewHandler(http.HandlerFunc(demoHandler), "otelhttp demo test" ) http.Handle("/v1/demo" , otelHandler) http.ListenAndServe(":6060" , nil ) } func demoGrpcReq () (string , error ) { conn, err := grpc.Dial("grpc-demo:1010" , grpc.WithInsecure(), grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()), grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()), ) if err != nil { return "" , err } defer conn.Close() client := demov1.NewGreeterClient(conn) resp, err := client.SayHello(context.TODO(), &demov1.HelloRequest{ Name: "http request" , }) if err != nil { return "" , err } return resp.GetMessage(), nil }
使用docker-compose up -d
启动演示服务,模拟请求curl 127.0.0.1:6060/v1/demo
,查看jaeger后台http://localhost:16686/
,示例如下:
总结
可观测
接入方式
Go版本
export方式
自定义标签
指标
原生promethues方式接入
支持范围广,本文用的1.14
promhttp
包
github.com/prometheus/client_golang/prometheus
包,WithLabelValues
指标
基于opentelemtry的promethues方式接入
版本太低报错,本文用的1.19
promhttp
包
go.opentelemetry.io/otel/exporters/prometheus
包,WithAttributes
可观测
接入方式
Go版本
主要依赖包
grpc中间件
追踪
原生jaeger方式接入
本文用的1.18
github.com/uber/jaeger-client-go
包,github.com/opentracing/opentracing-go
github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing
包,UnaryClientInterceptor
、WithTracer
追踪
基于opentelemtry的jaeger方式接入
版本太低报错,本文用的1.19
go.opentelemetry.io/otel/exporters/jaeger
包,go.opentelemetry.io/otel
包
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
包,UnaryClientInterceptor
、StreamClientInterceptor