K8s-kubectl(执行流程)

基于1.25

什么是kubectl

kubectl是kube-apiserver的客户端，通过向kube-apiserver发送HTTP请求，实现对K8s集群的控制

初始化命令对象

kubectl使用NewCmdXX 进行初始化，主要有俩类对象：

cobra.Command：提供命令对象
XXXoptions：Options命令参数对象

补全命令参数

支持补全成命令执行的相关工具类，形成完整的Options，主要工具包含三种

Genericclioptions.Recorder:用于记录资源对象发生变化的原因,对于由kubectl触发的变更，genericclioptions.Recoder会将完整的命令记录到资源对象的kubernetes.io/change-cause注解中
resource.QueryParamVerifier:用于检查资源对象是否支持DryRun和资源对象娇艳
options.ToPrinter/options.PrintObj:打印相关函数，用于获取命令执行结果之后，执行结果打印到终端

校验命令参数

依赖参数校验。比如，在使用kubectl apply 使用了--force-conflicts指定出现冲突的时候，合并更新，同时必须使用--service-side指定服务端应用
兼容参数校验。比如，在使用kubectl create的时候，使用了raw指定服务端的URL地址，就会执行使用Client把请求发送给服务端，不在支持使用–output定制结果输出

执行命令输出结果

校验命令参数之后，回进入到命令执行阶段。

其中在里面饮入了Builder和多层嵌套的Vistor模式，实现了不修改结果的情况下在现有的结果上添加新操作

定制资源构造器

resource.Builder 使用了Builder模式

Ref：https://github.com/kubernetes/kubectl/blob/6e4fe32a45fdcbf61e5c30ebdc511d75e7242432/pkg/cmd/util/factory.go#L41

// Factory provides abstractions that allow the Kubectl command to be extended across multiple types
// of resources and different API sets.
// The rings are here for a reason. In order for composers to be able to provide alternative factory implementations
// they need to provide low level pieces of *certain* functions so that when the factory calls back into itself
// it uses the custom version of the function. Rather than try to enumerate everything that someone would want to override
// we split the factory into rings, where each ring can depend on methods in an earlier ring, but cannot depend
// upon peer methods in its own ring.
// TODO: make the functions interfaces
// TODO: pass the various interfaces on the factory directly into the command constructors (so the
// commands are decoupled from the factory).
type Factory interface {
	genericclioptions.RESTClientGetter

	// DynamicClient returns a dynamic client ready for use
  // 实现看三种Client的交互模式
	DynamicClient() (dynamic.Interface, error)

	// KubernetesClientSet gives you back an external clientset
	KubernetesClientSet() (*kubernetes.Clientset, error)

	// Returns a RESTClient for accessing Kubernetes resources or an error.
	RESTClient() (*restclient.RESTClient, error)

	// NewBuilder returns an object that assists in loading objects from both disk and the server
	// and which implements the common patterns for CLI interactions with generic resources.
  // 实现Builder的对象实例化
	NewBuilder() *resource.Builder

	// Returns a RESTClient for working with the specified RESTMapping or an error. This is intended
	// for working with arbitrary resources and is not guaranteed to point to a Kubernetes APIServer.
	ClientForMapping(mapping *meta.RESTMapping) (resource.RESTClient, error)
	// Returns a RESTClient for working with Unstructured objects.
	UnstructuredClientForMapping(mapping *meta.RESTMapping) (resource.RESTClient, error)

	// Returns a schema that can validate objects stored on disk.
  // 提供了获取校验器对象的方法
	Validator(validationDirective string) (validation.Schema, error)

	// Used for retrieving openapi v2 resources.
	openapi.OpenAPIResourcesGetter

	// OpenAPIV3Schema returns a client for fetching parsed schemas for
	// any group version
	OpenAPIV3Client() (openapiclient.Client, error)
}

组装Vistor多层嵌套

Ref：https://github.com/kubernetes/kubectl/blob/3810c9e1663ed4b1a2eb64eb8d76f1c54de8a5c3/pkg/cmd/create/create.go#L259

r := f.NewBuilder().
	// 在发送请求到kube-apiserver到时候，使用Unstructured 非结构化数据
	Unstructured().
	// 配置对请求参数对象进行校验的校验器
	Schema(schema).
	// 知识在处理过程中出现错误不要立即结束流程，尽可能执行后面的流程
	ContinueOnError().
	// 指定暂存对象的命名空间，没有指定使用默认命名空间
	NamespaceParam(cmdNamespace).DefaultNamespace().
	// 配置输入参数重涉及到网络或者本地文件/目录
	FilenameParam(enforceNamespace, &o.FilenameOptions).
	// 指示在请求资源时使用标签选择器
	LabelSelectorParam(o.Selector).
	// 指示在处理请求数据遇到的嵌套列表数据展开并且拉平胃大列表
	Flatten().
	// 构造
	Do()

Vistor接口的相关定义：

Ref：https://github.com/kubernetes/cli-runtime/blob/777f84f55a05fe0cf98a21badfe6a0e39bf74553/pkg/resource/interfaces.go#L94
1
2
3
4
// Visitor lets clients walk a list of resources.
type Visitor interface {
Visit(VisitorFunc) error
}

Ref:https://github.com/kubernetes/cli-runtime/blob/777f84f55a05fe0cf98a21badfe6a0e39bf74553/pkg/resource/visitor.go#L63

// Info contains temporary info to execute a REST call, or show the results
// of an already completed REST call.
type Info struct {
	// Client will only be present if this builder was not local
	Client RESTClient
	// Mapping will only be present if this builder was not local
	Mapping *meta.RESTMapping

	// Namespace will be set if the object is namespaced and has a specified value.
	Namespace string
	Name      string

	// Optional, Source is the filename or URL to template file (.json or .yaml),
	// or stdin to use to handle the resource
	Source string
	// Optional, this is the most recent value returned by the server if available. It will
	// typically be in unstructured or internal forms, depending on how the Builder was
	// defined. If retrieved from the server, the Builder expects the mapping client to
	// decide the final form. Use the AsVersioned, AsUnstructured, and AsInternal helpers
	// to alter the object versions.
	// If Subresource is specified, this will be the object for the subresource.
	Object runtime.Object
	// Optional, this is the most recent resource version the server knows about for
	// this type of resource. It may not match the resource version of the object,
	// but if set it should be equal to or newer than the resource version of the
	// object (however the server defines resource version).
	ResourceVersion string
	// Optional, if specified, the object is the most recent value of the subresource
	// returned by the server if available.
	Subresource string
}

// Visit implements Visitor
func (i *Info) Visit(fn VisitorFunc) error {
	return fn(i, nil)
}

常见的Visitor实例用用途

Visitor	用途
DecoratedVistitor	提供多种装饰器函数，通过遍历自身的装饰器func 对Info和error处理，如果任何一个装饰函数报错立即返回
ContinueOnErrorVisitor	在执行多层Visitor匿名函数，如果发生错误不会马上退出，把错误收集到error数组中，返回
FlattenListVistor	将列表类型的通用资源展开为一个个独立的资源对象，并且试用其装饰的Vistor对象依次访问展开之后的资源对象
EagerVistor	Vistor集合类型，依次执行集合中的每个Vistor，同意收集错误为一个error信息返回
FileVistor	通过本地方式指定资源对象声明文件，会访问对象描述文件，并且通过StreamVistor将转换为Info
StreamVistitor	从io.Reader获取数据流，转换为JSON，通过schema检查，最后转为info对象

执行命令获取结果

Ref:https://github.com/kubernetes/kubectl/blob/3810c9e1663ed4b1a2eb64eb8d76f1c54de8a5c3/pkg/cmd/create/create.go#L274

err = r.Visit(func(info *resource.Info, err error) error {
		if err != nil {
			return err
		}
		if err := util.CreateOrUpdateAnnotation(cmdutil.GetFlagBool(cmd, cmdutil.ApplyAnnotationsFlag), info.Object, scheme.DefaultJSONEncoder()); err != nil {
			return cmdutil.AddSourceToErr("creating", info.Source, err)
		}

		if err := o.Recorder.Record(info.Object); err != nil {
			klog.V(4).Infof("error recording current command: %v", err)
		}

		if o.DryRunStrategy != cmdutil.DryRunClient {
			if o.DryRunStrategy == cmdutil.DryRunServer {
				if err := o.DryRunVerifier.HasSupport(info.Mapping.GroupVersionKind); err != nil {
					return cmdutil.AddSourceToErr("creating", info.Source, err)
				}
			}
			obj, err := resource.
				NewHelper(info.Client, info.Mapping).
				DryRun(o.DryRunStrategy == cmdutil.DryRunServer).
				WithFieldManager(o.fieldManager).
				WithFieldValidation(o.ValidationDirective).
				Create(info.Namespace, true, info.Object)
    }

打印执行结果

kubectl使用PrintObj func和ToPrinter func打印上一步获取的执行结果

Ref：https://github.com/kubernetes/cli-runtime/blob/777f84f55a05fe0cf98a21badfe6a0e39bf74553/pkg/printers/interface.go#L30

// ResourcePrinterFunc is a function that can print objects
type ResourcePrinterFunc func(runtime.Object, io.Writer) error

// PrintObj implements ResourcePrinter
func (fn ResourcePrinterFunc) PrintObj(obj runtime.Object, w io.Writer) error {
	return fn(obj, w)
}

// ResourcePrinter is an interface that knows how to print runtime objects.
type ResourcePrinter interface {
	// PrintObj receives a runtime object, formats it and prints it to a writer.
	PrintObj(runtime.Object, io.Writer) error
}

在补全命令阶段（Complete），根据传入打印参数的构造ResourcePrinter对象赋值为PrintObj，实现了（YAML、JSON）等打印命令执行结果

Ref:https://github.com/kubernetes/kubectl/blob/3810c9e1663ed4b1a2eb64eb8d76f1c54de8a5c3/pkg/cmd/annotate/annotate.go#L167


// Complete adapts from the command line args and factory to the data required.
func (o *AnnotateOptions) Complete(f cmdutil.Factory, cmd *cobra.Command, args []string) error {
	var err error

	o.RecordFlags.Complete(cmd)
	o.Recorder, err = o.RecordFlags.ToRecorder()
	if err != nil {
		return err
	}

	o.outputFormat = cmdutil.GetFlagString(cmd, "output")
	o.dryRunStrategy, err = cmdutil.GetDryRunStrategy(cmd)
	if err != nil {
		return err
	}
	dynamicClient, err := f.DynamicClient()
	if err != nil {
		return err
	}
	o.dryRunVerifier = resource.NewQueryParamVerifier(dynamicClient, f.OpenAPIGetter(), resource.QueryParamDryRun)

	cmdutil.PrintFlagsWithDryRunStrategy(o.PrintFlags, o.dryRunStrategy)
	printer, err := o.PrintFlags.ToPrinter()
	if err != nil {
		return err
	}
	o.PrintObj = func(obj runtime.Object, out io.Writer) error {
		return printer.PrintObj(obj, out)
	}

	if o.list && len(o.outputFormat) > 0 {
		return fmt.Errorf("--list and --output may not be specified together")
	}

	o.namespace, o.enforceNamespace, err = f.ToRawKubeConfigLoader().Namespace()
	if err != nil {
		return err
	}
	o.builder = f.NewBuilder()
	o.unstructuredClientForMapping = f.UnstructuredClientForMapping

	// retrieves resource and annotation args from args
	// also checks args to verify that all resources are specified before annotations
	resources, annotationArgs, err := cmdutil.GetResourcesAndPairs(args, "annotation")
	if err != nil {
		return err
	}
	o.resources = resources
	o.newAnnotations, o.removeAnnotations, err = parseAnnotations(annotationArgs)
	if err != nil {
		return err
	}

	return nil
}

ResourcePrinter	说明
YAMLPrinter	以YAML打印结果
JSONPrinter	以JSON打印结果
HuamanReadablePrinter	为指定输出格式默认使用Printer格式打印器，以可读性是打印执行结果，结果为表格
GoTempldatePrinter	按用户提供的GoTemplate执行打印结果
TablePrinter	转换为表格打印
CustomColumnsPrinter	按指定列名打印结果
SortingPrinter	按指定字段排序打印结果
JSONPathPrinter	按照用户提供的JSONpath打印结果
NamePrinter	为打印资源专门实现的Prnter，以rescue/name打印资源名称
EventPrinter	专门打印Event资源的Printer
OmitManageredFieldsPrinter	在打印资源忽略managerField字段