20210820 Update

20221205 Update

To-Do

• AZ-104
• AZ-204
• AZ-500
• AZ-305

References:

• https://learn.acloud.guru/series/certification-success-hub/view/azure-certification-guide
• https://learn.acloud.guru/series/certification-success-hub/view/azure-certification-guide-2021-eoy-update

The Azure AD Identity Platform

Managing Azure AD Tenant and Azure Subscription Associations

Azure AD Identities

Azure AD Groups

Azure AD Dynamic Groups

Azure AD Administrative Units

Hybrid Identity

Azure AD Connect and Hybrid Identities

Azure AD External Identities

User Flows with External Identities

Controlling Access

Securing Azure and Azure Active Directory (AD)

Azure Role-Based Access Control (RBAC)

Azure AD Roles

Custom Roles

Securing Identities and Access

Azure AD Privileged Identity Management (PIM)

Access Reviews

Azure AD Identity Protection

Conditional Access

Azure AD Passwordless Authentication

Securing Virtual Networks

Virtual Network Routing

Network S

Best Practices

Import

1. Turn on Query Caching for imported tables​
2. Turn on Large Dataset Storage format​

DirectQuery

1. Use star-schema for data model​and ensure that your dimension tables contain proper keys and that those keys relate to a fact table​
2. Materialize all aggregations, transformations, and calculations in SQL Server​
3. Pre-filter large fact tables in SQL Server​and remove filters in visual
   
   • Adding filters will cause Power BI to generate multiple SQL queries
4. Use dynamic M query parameter to move measure logic from Power BI to SQL Server​
5. Optimize measure definition to generate efficient DAX queries​
   
   • Power BI will translate complex DAX into multiple SQL Server queries
6. Use dual mode for dim tables​
   
   • Otherwise Power BI will use import dim tables to generate long SQL Queries for filters
7. Refine table structure and indexing strategy​
   
   • Clustered rowstore (like by TimeId) + non-clustered columnstore is recommended
8. Only

Compute

Virtual Machine

Azure Functions

App Services

Azure Container Instances (ACI)

Azure Kubernetes Service (AKS)

Azure Container Registry (ACR)

Windows Virtual Desktop

Networking

Virtual Network (VNet)

Network Security Group (NSG)

Application Security Group

Load Balancer

VPN Gateway

Application Gateway

ExpressRoute

Paired Region

Storage

Storage Account

Blob

Disk

File

Archive

Database

Cosmos DB

Azure SQL

Azure Database for PostgreSQL

Database Migration Services

Identity and Access Management (IAM)

Azure Active Directory (AAD)

Group

Role

Scope

Role-Based Access Control (RBAC)

Azure AD Join

Azure AD Connect

Single Sign-On

Self-Service Password Reset (SSPR)

I

S3

• Maximum object size is 5TB; largest object in a single PUT is 5GB.
• Recommended to use multi-part uploads if larger than 100MB.
• Security: IAM policies -> Bucket policy -> Object ACL
• 
• Versioning cannot be enabled at the object level. It's a bucket-level feature.

Amazon Neptune

Amazon Redshift

Amazon Athena

Amazon Quantum Ledger Database

Amazon Managed Blockchain

Amazon Timestream Database

Amazon DocumentDB

Amazon ElasticSearch

Databases

S3 Select vs Athena vs Redshift Spectrum

S3 Select is focused on retrieving data from S3 using SQL:

S3 Select, enables applications to retrieve only a subset of data from an object by using simple SQL expressions. By using S3 Select to retrieve only the data needed by your application, you can achieve drastic performance increases – in many cases you can get as much as a 400% improvement compared with classic S3

test.csv

key,a,b,c
a,1,,-1
a,2,,
a,3,,4

test.py

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession \
    .builder \
    .appName("spark-app") \
    .getOrCreate()
spark.sparkContext.setLogLevel("WARN")
df = spark.read.csv("test.csv", header=True)
res = df.groupBy(["key"]).agg(*[
    F.max("a"),
    F.max("b"),
    F.max("c"),
    F.min("a"),
    F.min("b"),
    F.min("c"),
])
print (res.toPandas())

spark-submit test.py

  key max(a) max(b) max(c) min(a) min(b) min(c)
0   a      3   None      4      1   None     -1

Compute

EC2

• Billing for interrupted Spot Instance
  
  
• When you launch an instance from AMI, it uses either paravirtual (PV) or hardware virtual machine (HVM) virtualization. HVM virtualization uses hardware-assist technology provided by the AWS platform.
• The information about the instance can be retrieved from:
  
  • http://169.254.169.254/latest/meta-data/
  • http://169.254.169.254/latest/user-data/
• The underlying Hypervisor for EC2:
  
  • Xen
  • Nitro
• Standard Reserved Instances cannot be moved between regions. You can choose if a Reserved Instance applies to either a specific AZ or an entire region, but you cannot change the region.
• About EC2 Auto Scaling
  
  • Can span multi-AZ
• About Placement Group
  
  • Three types of Placement Groups
    
    • Clustered Placement Group
      
      • Within a single AZ
      • Used for applications that need low network latency, high network throughput, or both
      • Only certain instances can be launched into a Clustered Placement Group
      • AWS r

Deploy kubeless to a Kubernetes cluster

$ export RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '"' -f 4)
$ kubectl create ns kubeless
$ kubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml

$ kubectl get pods -n kubeless
$ kubectl get deployment -n kubeless
$ kubectl get customresourcedefinition

Deploy sample function

def hello(event, context):
  print event
  return event['data']

$ kubeless function deploy hello --runtime python2.7 \
                                --from-file test.py \
                                --handler test.hello

$ kubectl get functions
$ kubeless function ls

$ kubeless function call hello --data 'Hello world!'

• create a file

  echo This is a sample text file > sample.txt

• delete a file

  del file_name

• move a file

  move stats.doc c:\statistics

• combine files

  copy /b file1 + file2 file3

import pandas as pd
import pyodbc
import sqlalchemy
import urllib
def get_sqlalchemy_engine(driver, server, uid, pwd, database):
    conn_str = 'DRIVER={};SERVER={};UID={};PWD={};DATABASE={}'.format(driver, server, uid, pwd, database)
    quoted = urllib.parse.quote_plus(conn_str)
    engine = sqlalchemy.create_engine('mssql+pyodbc:///?odbc_connect={}'.format(quoted))
    return engine
if __name__ == '__main__':
    # create engine
    driver = 'ODBC Driver 17 for SQL Server'
    server = 'xxx'
    uid = 'xxx'
    pwd = 'xxx'
    database = 'xxx'
    engine = get_sqlalchemy_engine(driver, server, uid, pwd, database)
    # read excel
    file_path = 'xxx'
    df = pd.read_excel(file_path)
    # load into SQL Server
    schema_name = 'xxx'
    table_name = 'xxx'
    df.to_sql(table_name, schema=schema_name, con=engine, index=False, if_exists='replace')

Steps

1. Create a network named "test"

   docker network create test

2. Create two containers using the network

   docker run --name c1 --network "test" --rm --entrypoint tail mongo -f
   docker run --name c2 --network "test" --rm --entrypoint tail mongo -f

3. Enter one container to ping the other and it will work

   docker exec -it c1 bash

   apt-get install iputils-ping  # install command ping

   root@79568c5ce391:/usr/src/app# ping c2
   PING c2 (172.18.0.3) 56(84) bytes of data.
   64 bytes from c2.test (172.18.0.3): icmp_seq=1 ttl=64 time=0.137 ms
   64 bytes from c2.test (172.18.0.3): icmp_seq=2 ttl=64 time=0.221 ms
   64 bytes from c2.test (172.18.0.3): icmp_seq=3 ttl=64 time=0.232 ms
   ...

Notes

Using default network or "bridge" network does not work:

docker run --name c1 --rm --entrypoint tail web_scraper:v1 -f
docker run --name c2 --rm --entrypoint tail web_scraper:v1 -f

docker run --name c1 --network "bridge" --rm --entrypoint tail web_scraper:v1 

$ docker run -p 127.0.0.1:80:8080/tcp ubuntu bash

This binds port 8080 of the container to TCP port 80 on 127.0.0.1 of the host machine. You can also specify udp and sctp ports.

$ docker run --expose 80 ubuntu bash

This exposes port 80 of the container without publishing the port to the host system’s interfaces.

Images

docker images
docker build -t image_name .
docker rmi $(docker images | grep "^<none>" | awk "{print $3}")  # remove all untagged images
docker save image_name > image_name.tar  # save image as a tar file
docker load < busybox.tar.gz  # load image

Containers

docker run -p 27017:27017 -v mongodbdata:/data/db mongo
docker ps -a
docker exec -it ubuntu_bash bash
docker rm container_name
docker rm $(docker ps -a -q)  # remove all stopped containers

Volumes

docker volume create mongodbdata
docker volume ls
docker volume inspect mongodbdata

Networks

docker network ls
docker network create network_name
docker network inspect network_name
docker network rm network_name

Azure Container Instance

docker login azure --tenant-id 8432da0f-f8af-4b02-b318-4c777cfab498
docker context create aci hpjacicontext
docker context use hpjacicontext
docker compose up

Sample Code: https://github.com/guangningyu/react-skeleton