Guangning Yu's Blog

Kubeless Basics

2019-05-14 02:09:51  |  Kubernetes

Deploy kubeless to a Kubernetes cluster

  1. $ export RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '"' -f 4)
  2. $ kubectl create ns kubeless
  3. $ kubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml
  1. $ kubectl get pods -n kubeless
  2. $ kubectl get deployment -n kubeless
  3. $ kubectl get customresourcedefinition

Deploy sample function

  1. def hello(event, context):
  2. print event
  3. return event['data']
  1. $ kubeless function deploy hello --runtime python2.7 \
  2. --from-file test.py \
  3. --handler test.hello
  1. $ kubectl get functions
  2. $ kubeless function ls
  1. $ kubeless function call hello --data 'Hello world!'

Windows cmd

2019-03-19 10:31:36  |  Windows
  • create a file

    1. echo This is a sample text file > sample.txt
  • delete a file

    1. del file_name
  • move a file

    1. move stats.doc c:\statistics
  • combine files

    1. copy /b file1 + file2 file3

Load Excel file into SQL Server

2019-03-07 07:57:56  |  Python
  1. import pandas as pd
  2. import pyodbc
  3. import sqlalchemy
  4. import urllib
  5. def get_sqlalchemy_engine(driver, server, uid, pwd, database):
  6. conn_str = 'DRIVER={};SERVER={};UID={};PWD={};DATABASE={}'.format(driver, server, uid, pwd, database)
  7. quoted = urllib.parse.quote_plus(conn_str)
  8. engine = sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(quoted))
  9. return engine
  10. if __name__ == '__main__':
  11. # create engine
  12. driver = 'ODBC Driver 17 for SQL Server'
  13. server = 'xxx'
  14. uid = 'xxx'
  15. pwd = 'xxx'
  16. database = 'xxx'
  17. engine = get_sqlalchemy_engine(driver, server, uid, pwd, database)
  18. # read excel
  19. file_path = 'xxx'
  20. df = pd.read_excel(file_path)
  21. # load into SQL Server
  22. schema_name = 'xxx'
  23. table_name = 'xxx'
  24. df.to_sql(table_name, schema=schema_name, con=engine, index=False, if_exists='replace')

Setup SQL Server and pyodbc

2019-03-05 02:59:13

Setup SQL Server

Connect pyodbc to SQL Server

  1. import pyodbc
  2. conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=test;DATABASE=test;UID=user;PWD=password')
  3. cursor = conn.cursor()
  4. for row in cursor.tables():
  5. print(row.table_name)

Access another container in Docker

2019-02-26 06:28:04  |  Docker

Steps

  1. Create a network named "test"

    1. docker network create test
  2. Create two containers using the network

    1. docker run --name c1 --network "test" --rm --entrypoint tail mongo -f
    2. docker run --name c2 --network "test" --rm --entrypoint tail mongo -f
  3. Enter one container to ping the other and it will work

    1. docker exec -it c1 bash
    1. apt-get install iputils-ping # install command ping
    1. root@79568c5ce391:/usr/src/app# ping c2
    2. PING c2 (172.18.0.3) 56(84) bytes of data.
    3. 64 bytes from c2.test (172.18.0.3): icmp_seq=1 ttl=64 time=0.137 ms
    4. 64 bytes from c2.test (172.18.0.3): icmp_seq=2 ttl=64 time=0.221 ms
    5. 64 bytes from c2.test (172.18.0.3): icmp_seq=3 ttl=64 time=0.232 ms
    6. ...

Notes

Using default network or "bridge" network does not work:

  1. docker run --name c1 --rm --entrypoint tail web_scraper:v1 -f
  2. docker run --name c2 --rm --entrypoint tail web_scraper:v1 -f
  1. docker run --name c1 --network "bridge" --rm --entrypoint tail web_scraper:v1

Publish or expose port (-p, --expose)

2019-02-26 05:42:45  |  Docker
  1. $ docker run -p 127.0.0.1:80:8080/tcp ubuntu bash

This binds port 8080 of the container to TCP port 80 on 127.0.0.1 of the host machine. You can also specify udp and sctp ports.

  1. $ docker run --expose 80 ubuntu bash

This exposes port 80 of the container without publishing the port to the host system’s interfaces.

Fix Chinese characters won't display in SSH

2019-02-24 15:29:55

Solution

Set up the same locale in both of the local laptop and remoter server:

  1. export LC_ALL=en_US.UTF-8
  2. export LANG=en_US.UTF-8

Expand the EBS root volume of EC2 Linux instance

2019-02-24 07:10:42  |  AWS
  1. Modify the EBS Volume from the console https://console.aws.amazon.com/ec2/
  2. Use the lsblk command to list the block devices attached to the instance

    1. $ lsblk
    2. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
    3. xvda 202:0 0 30G 0 disk
    4. `-xvda1 202:1 0 8G 0 part /
    5. loop0 7:0 0 91M 1 loop /snap/core/6405
    6. loop1 7:1 0 87.9M 1 loop /snap/core/5742
    7. loop2 7:2 0 17.9M 1 loop /snap/amazon-ssm-agent/1068
    8. loop3 7:3 0 16.5M 1 loop /snap/amazon-ssm-agent/784
    9. loop4 7:4 0 18M 1 loop /snap/amazon-ssm-agent/930
  3. Use the df -h command to report the existing disk space usage on the file system

    1. $ sudo df -h /dev/xvd*
    2. Filesystem Size Used Avail Use% Mounted on
    3. udev 488M 0 488M 0% /dev
    4. /dev/xvda1 7.7G 7.4G 370M 96% /
  4. Expand the modified partition using growpart

    1. $ sudo growpart /dev/xvda 1
    2. CHANGED: partition=1 start=2048 old: size=16775135 end=16777183 new: size=62912479,end=62914

Create Free SSL/TLS Certificates using Certbot

2019-02-24 04:50:01

Install

  1. sudo add-apt-repository ppa:certbot/certbot
  2. sudo apt-get update
  3. sudo apt-get install certbot python-certbot-nginx

Setup

Running this command will get a certificate for you and have Certbot edit your Nginx configuration automatically to serve it.

  1. sudo certbot --nginx

Renew

The Certbot packages on your system come with a cron job that will renew your certificates automatically before they expire.
You can test automatic renewal for your certificates by running this command:

  1. sudo certbot renew --dry-run

Docker Basics

2019-02-20 14:04:31  |  Docker

Images

  1. docker images
  2. docker build -t image_name .
  3. docker rmi $(docker images | grep "^<none>" | awk "{print $3}") # remove all untagged images
  4. docker save image_name > image_name.tar # save image as a tar file
  5. docker load < busybox.tar.gz # load image

Containers

  1. docker run -p 27017:27017 -v mongodbdata:/data/db mongo
  2. docker ps -a
  3. docker exec -it ubuntu_bash bash
  4. docker rm container_name
  5. docker rm $(docker ps -a -q) # remove all stopped containers

Volumes

  1. docker volume create mongodbdata
  2. docker volume ls
  3. docker volume inspect mongodbdata

Networks

  1. docker network ls
  2. docker network create network_name
  3. docker network inspect network_name
  4. docker network rm network_name

Webpack4 Setup

2019-02-12 15:42:38  |  Javascript

Kubernetes Basics

2019-02-16 17:40:29  |  Kubernetes

Graph

2019-01-28 10:01:15
### Graph Databases - Graphql - Apache Giraph - Apache GraphX - Neo4j - Titan - OrientDB ### Graph Algorithms - PageRank - Connected Components - Triangle Counting ### Graph Theory - Vertex - Edge

test jms

2019-02-16 17:40:29
  1. set proxy
  1. wget http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.rpm
    3.

Building Java Projects with Maven

2019-02-16 17:40:29  |  Java Maven

Create the directory structure

  1. mkdir -p src/main/java/hello
  2. mkdir -p src/test/java/hello

Create classes

src/main/java/hello/HelloWorld.java

  1. package hello;
  2. import org.joda.time.LocalTime;
  3. public class HelloWorld {
  4. public static void main(String[] args) {
  5. LocalTime currentTime = new LocalTime();
  6. System.out.println("The current local time is: " + currentTime);
  7. Greeter greeter = new Greeter();
  8. System.out.println(greeter.sayHello());
  9. }
  10. }

src/main/java/hello/Greeter.java

  1. package hello;
  2. public class Greeter {
  3. public String sayHello() {
  4. return "Hello world!";
  5. }
  6. }

Write a test

src/test/java/hello/GreeterTest.java

  1. package hello;
  2. import static org.hamcrest.CoreMatchers.containsString;
  3. import static org.junit.Assert.*;
  4. import org.junit.Test;
  5. public class GreeterTest {
  6. private Greeter greeter = new Greeter();
  7. @Test
  8. public void greeterSaysHello() {
  9. assertThat(greeter.sayHel

Crontab basics

2019-02-16 17:40:29
  • every 30 minutes
  1. 0,30 * * * * ...

Plot decay function

2019-02-16 17:40:32
```python import matplotlib.pyplot as plt import numpy as np alpha = [0.99, 0.9, 0.8] x = np.arange(0, 100, 1) y1 = np.vectorize(lambda t: alpha[0]**t)(x) y2 = np.vectorize(lambda t: alpha[1]**t)(x)

Lodash Basics

2019-02-16 17:40:29
  • loop through object
  1. > var _ = require('lodash');
  2. > foo
  3. { a: { num: 1 }, b: { num: 2 }, c: { num: 3 } }
  4. > _.forEach(foo, (vlu, key) => {console.log(vlu, key);})
  5. { num: 1 } 'a'
  6. { num: 2 } 'b'
  7. { num: 3 } 'c'
  8. { a: { num: 1 }, b: { num: 2 }, c: { num: 3 } }
  • filter object of objects
  1. > foo
  2. { a: { num: 1 }, b: { num: 2 }, c: { num: 3 } }
  3. > _.pickBy(foo, (i) => i.num>=2)
  4. { b: { num: 2 }, c: { num: 3 } }

Setup Nginx on Ubuntu 14.04

2018-11-21 02:00:54
- start `nginx` service ``` sudo service apache2 stop sudo service nginx start sudo service nginx status ``` - restart `nginx` when the server is rebooted ``` sudo update-rc.d nginx defaults ``` -

Bash Basics

2019-02-16 17:40:29  |  shell
- assign default value to variable ``` DEFAULT=5 RESULT=${VAR:-$DEFAULT} ``` - trap error exit ``` function rollback() { if [[ $? -ne 0 ]]; then echo "rollback" fi } set -ueo pipefail trap r