Going towards microservice architecture, you should be aware of complexity that it brings. Multiplication of replicas causes your infrastructure harder to monitor and maintain. You are simply not capable of keep an eye of each service by yourself. Additionally Kubernetes (k8s) infrastructure by itself must be monitored.
On the Market there are many tools which can help us with monitoring. Some of them are paid, others are open source. But thinking about monitoring kubernetes the most recommended one is Prometheus with combination with Alert Manager and Grafana.
Setup for the testing is a quite easy task, but for sure production release of those tools are more complicated. But definitely it pays off.
There are many free to use dashboards and alerts templates. After a while even creating your own dashboard is rather a quite easy task.
Here are some recommendations:
1) Dashboards are really nice to eye, but overcomplicated can be hard to read. You rather should keep your board as simple as green/red switch. Be sure that your admins will not be looking on dashboard all day. But red color should be easily noticeable.
Above is a rather gently captured sentence “Dashboards are useless! They are a waste of time. Get Netflix if you want to watch something. It’s cheaper than any other option.” – Viktor Farcic, the devops 2.5 toolkit – Monitoring, Logging and Auto-Scaling Kubernetes.
2) Alerting is crucial in your monitoring infrastructure. Good alerts adjustment is a key – to many alerts will be (after a while) ignored by receiver group, to few can cause outage of your infrastructure. Adjust the receiver group as well, choose people who are really able to do something (apart of acting as an proxy) in this particular matter.
3) Reconfiguring those tools after changing something in configuration can be very painful. In that case you have to recreate your monitoring infrastructure or configuring it on web interface. None of these solutions are devops friendly.
Consider using Prometheus Operator instead. This is a declarative way of defining your monitoring infrastructure. It was initially created by CoreOS team and now is under RedHat maintenance.
You gain, practically out of the box, monitoring of each service in k8s. Additionally you can easily add your own metric endpoints, dashboards, alert definitions.
4) You can consider adding your own metrics in your applications (https://prometheus.io/docs/instrumenting/clientlibs/) . Often this is a better idea, then storing all information in logs.